Cmos image sensor for rgb imaging and depth measurement with laser sheet scan

ABSTRACT

An imaging unit includes a light source and a pixel array. The light source projects a line of light that is scanned in a first direction across a field of view of the light source. The line of light oriented in a second direction that is substantially perpendicular to the first direction. The pixel array is arranged in at least one row of pixels that extends in a direction that is substantially parallel to the second direction. At least one pixel in a row is capable of generating two-dimensional color information of an object in the field of view based on a first light reflected from the object and is capable of generating three-dimensional (3D) depth information of the object based on the line of light reflecting from the object. The 3D-depth information includes time-of-flight information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part patent application of U.S.patent application Ser. No. 16/191,415, filed Nov. 14, 2018, and acontinuation-in-part application of U.S. patent application Ser. No.16/186,477, filed Nov. 9, 2018, which are both continuation patentapplications of U.S. patent application Ser. No. 14/842,825, filed Sep.1, 2015, now U.S. Pat. No. 10,132,616, which claims the priority benefitunder 35 U.S.C. § 119(e) of U.S. Provisional Application No. 62/182,404filed on Jun. 19, 2015, and U.S. Provisional Application No. 62/150,252filed on Apr. 20, 2015, the disclosures of each are incorporated hereinby reference in their entireties. Additionally, the present patentapplication claims the priority benefit under 35 U.S.C. § 119(e) ofProvisional Application No. 62/783,164 filed on Dec. 20, 2018, thedisclosure of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to image sensors.More specifically, and not by way of limitation, particular embodimentsof the subject matter disclosed herein are directed to a complementarymetal oxide semiconductor (CMOS) image sensor in which each pixel of theimage sensor may be used for two-dimensional (2D) imaging as well aspoint-scan based and sheet-scan based three-dimensional (3D) depthmeasurements with ambient light rejection.

BACKGROUND

Three-dimensional imaging systems are increasingly being used in a widevariety of applications such as, industrial production, video games,computer graphics, robotic surgeries, consumer displays, surveillancevideos, 3D modeling, real estate sales, and so on. Existing 3D imagingtechnologies may include, for example, time-of-flight (TOF) based rangeimaging, stereo vision systems, and structured light (SL) methods.

In a TOF-based range imaging, distance to a 3D object may be resolvedbased on the known speed of light by measuring the round-trip time ittakes for a light signal to travel between a camera and the 3D objectfor each point of the image. A TOF camera may use a scannerless approachto capture the entire scene with each laser or light pulse. Some exampleapplications of the TOF-base range imaging may include advancedautomotive applications, such as active pedestrian safety or pre-crashdetection based on distance images in real time; to track movements ofhumans, such as during interaction with games on video game consoles; inindustrial machine vision to classify objects and help robots find theitems, such as items on a conveyor belt, and so on.

In stereoscopic imaging or stereo vision systems, two cameras displacedhorizontally from one another are used to obtain two differing views ona scene or a 3D object in the scene. By comparing these two images,relative depth information may be obtained for the 3D object. Stereovision is highly important in fields, such as robotics, to extractinformation about the relative position of 3D objects in the vicinity ofautonomous systems and/or robots. Other stereo-vision applications forrobotics include object recognition in which stereoscopic depthinformation allows a robotic system to separate occluding imagecomponents that a robot may otherwise not be able to distinguish as twoseparate objects, such as one object in front of another, partially orfully hiding the other object. Three-dimensional stereo displays arealso used in entertainment and automated systems.

In an SL approach, the 3D shape of an object may be measured usingprojected light patterns and a camera for imaging. A known pattern oflight (often grids or horizontal bars or patterns of parallel stripes)is projected onto a scene or a 3D object in the scene. The projectedpattern may become deformed or displaced when striking the surface ofthe 3D object. Such deformation may allow an SL vision system tocalculate the depth and surface information of the object. Thus,projecting a narrow band of light onto a 3D surface may produce a lineof illumination that may appear distorted from other perspectives thanthat of the projector, and can be used for geometric reconstruction ofthe illuminated surface shape. An SL-based 3D-imaging technique maybeused in different applications such as, by a police force to photographfingerprints in a 3D scene, inline inspection of components during aproduction process, in health care for live measurements of human bodyshapes or the micro structures of human skin, and the like.

SUMMARY

An example embodiment provides an imaging unit that may include a lightsource and a pixel array. The light source may project a line of lightthat is scanned in a first direction across a field of view of the lightsource. The line of light may be oriented in a second direction that issubstantially perpendicular to the first direction. The pixel array maybe arranged in at least one row of pixels that extends in a directionthat is substantially parallel to the second direction. At least onepixel in a row may be capable of generating two-dimensional (2D) colorinformation of an object in the field of view of the light source basedon a first light reflected from the object and capable of generatingthree-dimensional (3D) depth information of the object based on the lineof light reflecting from the object. The 3D-depth information mayinclude time-of-flight information. In one embodiment, the imaging unitmay include a time-to-digital converter coupled to the pixel in whichthe time-to-digital converter may generate the 3D-depth informationbased on the pixel detecting the line of light being reflected from theobject. The 3D-depth information may include timestamp information.

An example embodiment provides an image sensor unit that may include apixel array and a time-to-digital converter. The pixel array may bearranged in at least one row of pixels that extends in a firstdirection. At least one pixel in a row may be capable of generatingtwo-dimensional (2D) color information of an object based on a firstlight reflected from the object in a field of view of the pixel arrayand capable of generating 3D-depth information of the object based on aline of light reflecting from the object. The 3D-depth information mayinclude time-of-flight information. The light of light may be orientedin a second direction that is substantially perpendicular to the firstdirection, and the line of light may be scanned across the field of viewof the pixel array in substantially the first direction. Thetime-to-digital converter may be coupled to the pixel, and may generatethe 3D-depth information based on the pixel detecting the line of lightbeing reflected from the object. In one embodiment, the image sensor mayinclude a plurality of time-to-digital converters. Each pixel in a rowof the pixel array may be coupled to a corresponding time-to-digitalconverter that generates the 3D-depth information for the pixel based onthe pixel detecting the line of light being reflected from the object.

An example embodiment provides a method that may include: projectingfrom a light source a line of light oriented in a first direction acrossa field of view of a light source in a second direction that issubstantially perpendicular to the first direction; and generating at apixel two-dimensional (2D) color information of an object in the fieldof view of the light source based on a first light reflected from theobject and three-dimensional (3D) depth information of the object basedon the line of light reflecting from the object, the pixel being capableof generating 2D color information of the object and 3D-depthinformation of the object, the pixel further being part of a pixel arraythat is arranged in at least one row of pixels that extends in adirection that is substantially parallel to the second direction, thepixel being in a row of the pixel array, and the 3D-depth informationcomprising time-of-flight information.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following section, the inventive aspects of the presentdisclosure will be described with reference to exemplary embodimentsillustrated in the figures, in which:

FIG. 1 depicts a highly simplified, partial layout of a system accordingto one embodiment of the subject matter disclosed herein;

FIG. 2 depicts an example operational layout of the system depicted inFIG. 1 according to the subject matter disclosed herein;

FIG. 3 is an example depiction of how a point scan may be performed for3D-depth measurements according to the subject matter disclosed herein;

FIG. 4 depicts an exemplary time-stamping for scanned light spotsaccording to the subject matter disclosed herein;

FIG. 5 depicts example circuit details of the 2D pixel array and aportion of the associated processing circuits in the image processingunit of the image sensor depicted in FIGS. 1 and 2 according to thesubject matter disclosed herein;

FIG. 6A depicts an exemplary layout of an image sensor unit, such as theimage sensor unit depicted in FIG. 5, according to one embodiment of thesubject matter disclosed herein;

FIG. 6B depicts architectural details of an example correlated doublesample and analog-to-digital converter unit for 3D-depth measurementaccording to one embodiment of the subject matter disclosed herein;

FIG. 7 depicts a timing diagram that shows example timing of differentsignals in the system of FIGS. 1 and 2 to generate timestamp-basedpixel-specific outputs in a 3D-linear mode of operation according toparticular embodiments of the subject matter disclosed herein;

FIG. 8 depicts an example look-up table to show how a look-up table maybe used in particular embodiments disclosed herein to determine 3D-depthvalues;

FIG. 9 depicts an exemplary flowchart showing how the same image sensor,such as the image sensor unit in FIGS. 1 and 2, may be used for both 2Dimaging and 3D-depth measurements according to the subject matterdisclosed herein;

FIG. 10 depicts a timing diagram that shows example timing of differentsignals in the system of FIGS. 1 and 2 to generate a 2D image using a2D-linear mode of operation according to the subject matter disclosedherein;

FIG. 11 depicts a timing diagram that shows example timing of differentsignals in the system of FIGS. 1 and 2 to generate timestamp-basedpixel-specific outputs in a 3D-logarithmic (log) mode of operationaccording to the subject matter disclosed herein;

FIG. 12 depicts another example embodiment of an image sensor that maymake 3D-depth measurements using a sheet scan according to the subjectmatter disclosed herein;

FIG. 13 depicts an example LUT that may be used to determine 3D-depthvalues for a sheet scan according to the subject matter disclosedherein;

FIG. 14 depicts a block diagram of an example embodiment of a pixelarray and of associated processing circuits according to the subjectmatter disclosed herein;

FIG. 15 depicts an example embodiment of the pixel array and theassociated processing circuits of FIG. 12 according to the subjectmatter disclosed herein; and

FIG. 16 depicts an example overall layout of the system depicted inFIGS. 1 and 2 according to the subject matte disclosed herein.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the disclosure. Itwill, however, be understood by those skilled in the art that thedisclosed inventive aspects may be practiced without these specificdetails. In other instances, well-known methods, procedures, componentsand circuits have not been described in detail so as not to obscure thepresent disclosure. Additionally, the described inventive aspects can beimplemented to perform low power, 3D depth measurements in any imagingdevice or system, including, for example, a smartphone, a User Equipment(UE), a laptop computer, and the like.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present disclosure. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” or“according to one embodiment” (or other phrases having similar import)in various places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments. Also, depending on the context of descriptionherein, a singular term may include its plural forms and a plural termmay include its singular form. Similarly, a hyphenated term (e.g.,“two-dimensional,” “pre-determined,” “pixel-specific,” etc.) may beoccasionally interchangeably used with its non-hyphenated version (e.g.,“two dimensional,” “predetermined,” “pixel specific,” etc.), and acapitalized entry (e.g., “Counter Clock,” “Row Select,” “PIXOUT,” etc.)may be interchangeably used with its non-capitalized version (e.g.,“counter clock,” “row select,” “pixout,” etc.). Such occasionalinterchangeable uses shall not be considered inconsistent with eachother.

It is noted at the outset that the terms “coupled,” “operativelycoupled,” “connected,” “connecting,” “electrically connected,” etc., maybe used interchangeably herein to generally refer to the condition ofbeing electrically/electronically connected in an operative manner.Similarly, a first entity is considered to be in “communication” with asecond entity (or entities) when the first entity electrically sendsand/or receives (whether through wireline or wireless means) informationsignals (whether containing address, data, or control information)to/from the second entity regardless of the type (analog or digital) ofthose signals. It is further noted that various figures (includingcomponent diagrams) shown and described herein are for illustrativepurpose only, and are not drawn to scale. Similarly, various waveformsand timing diagrams are shown for illustrative purpose only.

The terms “first,” “second,” etc., as used herein, are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.) unless explicitly defined assuch. Furthermore, the same reference numerals may be used across two ormore figures to refer to parts, components, blocks, circuits, units, ormodules having the same or similar functionality. Such usage is,however, for simplicity of illustration and ease of description only; itdoes not imply that the construction or architectural details of suchcomponents or units are the same across all embodiments or suchcommonly-referenced parts/modules are the only way to implement theteachings of particular embodiments of the present disclosure.

As used herein, the term “module” refers to any combination of software,firmware and/or hardware configured to provide the functionalitydescribed herein in connection with a module. The software may beembodied as a software package, code and/or instruction set orinstructions, and the term “hardware,” as used in any implementationdescribed herein, may include, for example, singly or in anycombination, hardwired circuitry, programmable circuitry, state machinecircuitry, and/or firmware that stores instructions executed byprogrammable circuitry. The modules may, collectively or individually,be embodied as circuitry that forms part of a larger system, forexample, but not limited to, an integrated circuit (IC), system on-chip(SoC) and so forth.

The earlier-mentioned 3D technologies may have many drawbacks. Forexample, a TOF-based 3D-imaging system may require high power to operateoptical or electrical shutters. Such TOF-based 3D-imaging systemstypically operate over a range of few meters to several tens of meters,but the resolution of such systems decreases for measurements over shortdistances, thereby making 3D imaging within a distance of about onemeter almost impractical. Hence, a TOF-based system may not be desirablefor cell phone-based camera applications in which pictures arepre-dominantly taken at close distances. A TOF sensor may also requirespecial pixels having big pixel sizes, usually larger than 7 μm. Thesepixels also may be vulnerable to ambient light.

The stereoscopic imaging approach generally works only with texturedsurfaces. It has high computational complexity because of the need tomatch features and find correspondences between the stereo pair ofimages of an object. This requires high system power, which is not adesirable attribute if power conservation is needed, such as insmartphones. Furthermore, stereo imaging requires two regular, high bitresolution sensors along with two lenses, making the entire assemblyunsuitable for applications in portable devices, like cell phones ortablets, in which device real estate is at a premium.

An SL approach introduces distance ambiguity, and also requires highsystem power. For 3D-depth measurements, the SL method may need multipleimages with multiple patterns, all of which increase computationalcomplexity and power consumption. Furthermore, the SL imaging may alsorequire regular image sensors with high bit resolution. Thus, astructured light-based system may not be suitable for low-cost,low-power, compact image sensors in smartphones.

In contrast to the above-mentioned 3D technologies, particularembodiments of the present disclosure provide for implementing a lowpower, 3D-imaging system on portable electronic devices, such assmartphones, tablets, UEs, and the like. A 2D-imaging sensor as perparticular embodiments of the present disclosure can capture both 2D RGB(Red, Green, Blue) images and 3D-depth measurements with visible lightlaser scanning, while being able to reject ambient light during 3D-depthmeasurements. It is noted here that although the following descriptionmay frequently mention the visible light laser as a light source forpoint-scans or sheet scans, and a 2D RGB sensor as an image/lightcapture device, such mention is for the purpose of consistency ofdescription only. The visible laser and RGB sensor based examplesdescribed below may find applications in low-power, consumer-grademobile electronic devices with cameras such as, smartphones, tablets, orUEs. It is, however, understood that the subject matter disclosed hereinis not limited to the visible laser-RGB sensor based examples mentionedbelow. Rather, according to particular embodiments of the subject matterdisclosed herein, the point scan-based 3D-depth measurements and theambient light rejection methodology may be performed using manydifferent combinations of 2D sensors and laser light sources (for pointscans and/or for sheet scans), such as: (i) a 2D color (RGB) sensor witha visible light laser source, in which the laser source may be a red,green, or blue light laser, or a laser source producing a combination ofthese colored lights; (ii) a visible light laser with a 2D RGB colorsensor having an Infrared (IR) cut filter; (iii) a Near Infrared (NIR)laser with a 2D IR sensor; (iv) an NIR laser with a 2D NIR sensor; (v)an NIR laser with a 2D RGB sensor (without an IR cut filter); (vi) anNIR laser with a 2D RGB sensor (without an NIR cut filter); (vii) a 2DRGB-IR sensor with visible or NIR laser; (viii) a 2D RGBW (red, green,blue, white) sensor with either visible or NIR laser; and so on.

During 3D-depth measurements, the entire sensor may operate as a binarysensor in conjunction with the laser scan to reconstruct 3D content. Inparticular embodiments, the pixel size of the sensor may be as small as1 μm. Furthermore, due to lower bit resolution, analog-to-digitalconverter (ADC) units in the image sensor according to the subjectmatter disclosed herein may require significantly less processing powerthan that is needed for high-bit resolution sensors in traditional3D-imaging systems. Because of the need for less processing power, a3D-imaging module according to the subject matter disclosed herein mayrequire low system power and, hence, may be quite suitable for inclusionin low power devices like smartphones.

In particular embodiments, the subject matter disclosed herein may usetriangulation and point scans with a laser light source for 3D-depthmeasurements with a group of sensors in a line. The laser scanning planeand the imaging plane may be oriented using epipolar geometry. An imagesensor according to one embodiment of the subject matter disclosedherein may use timestamps to remove ambiguity in the triangulationapproach, thereby reducing the amount of depth computations and systempower. The same image sensor, that is, each pixel in the image sensor,may be used in the normal 2D (RGB color or non-RGB) imaging mode as wellas in the 3D laser-scan modes. In the laser-scan mode (i.e., a pointscan or a sheet scan), however, the resolution of the ADCs in the imagesensor may be reduced to a binary output (1-bit resolution only),thereby improving the readout speed and reducing power consumption dueto, for example, switching in the ADC units, in the chip incorporatingthe image sensor and associated processing units. Furthermore, thepoint-scan approach and the sheet-scan approach may allow the system totake all measurements in one pass, thereby reducing the latency fordepth measurements and reducing motion blur.

As noted before, in particular embodiments, the entire image sensor maybe used for routine 2D RGB color imaging using, for example, ambientlight, as well as for 3D-depth imaging using visible laser scan. Suchdual use of the same camera unit may save space and cost for mobiledevices. Furthermore, in certain applications, the user of visible laserfor 3D applications may be better for eye safety of a user as comparedto a near infrared (NIR) laser. The sensor may have higher quantumefficiency at visible spectrum that at the NIR spectrum, leading tolower power consumption of the light source. In one embodiment, thedual-use image sensor may work in a linear mode of operation for 2Dimaging as a regular 2D sensor. For 3D imaging, however, the sensor maywork in a linear mode under moderate lighting condition and in alogarithmic mode under strong ambient light to facilitate continued useof the visible laser source through rejection of the strong ambientlight. Furthermore, ambient light rejection may be needed in case of anNIR laser as well, for example, when the bandwidth of the pass band ofan IR-cut filter employed with an RGB sensor is not narrow enough.

FIG. 1 depicts a highly simplified, partial layout of a system 15according to one embodiment of the subject matter disclosed herein. Asdepicted, the system 15 may include an imaging module 17 coupled to andin communication with a processor or host 19. The system 15 may alsoinclude a memory module 20 coupled to the processor 19 to storeinformation content such as, image data received from the imaging module17. In particular embodiments, the entire system 15 may be encapsulatedin a single integrated circuit (IC) or chip. Alternatively, each of themodules 17, 19, and 20 may be implemented in a separate chip.Furthermore, the memory module 20 may include more than one memory chip,and the processor module 19 may include multiple processing chips aswell. In any event, the details about packaging of the modules in FIG. 1and how the modules are fabricated or implemented in a single chip orusing multiple discrete chips are not relevant to the present disclosureand, hence, such details are not provided herein.

The system 15 may be any low power, electronic device configured for 2D-and 3D-camera applications according to the subject matter disclosedherein. The system 15 may be portable or non-portable. Some examples ofthe portable version of the system 15 may include popular consumerelectronic gadgets such as, a mobile device, a cellphone, a smartphone,a user equipment (UE), a tablet, a digital camera, a laptop or desktopcomputer, an electronic smartwatch, a machine-to-machine (M2M)communication unit, a virtual reality (VR) equipment or module, a robot,and the like. On the other hand, some examples of the non-portableversion of the system 15 may include a game console in a video arcade,an interactive video terminal, an automobile, a machine vision system,an industrial robot, a VR equipment, a driver-side mounted camera in acar (for example, to monitor whether the driver is awake), and so on.The 3D-imaging functionality provided according to the subject matterdisclosed herein may be used in many applications such as, virtualreality applications on a virtual reality equipment, onlinechatting/gaming, 3D texting, searching an online or local (device-based)catalog/database using a 3D image of an item to obtain informationrelated to the item (for example, calorie content of a piece of fooditem), robotics and machine vision applications, automobileapplications, such as autonomous driving applications, and the like.

In particular embodiments disclosed herein, the imaging module 17 mayinclude a light source 22 and an image sensor unit 24. As described inmore detail with reference to FIG. 2 below, in one embodiment the lightsource 22 may be a visible laser. In other embodiments, the light sourcemay be an NIR laser. The image sensor unit 24 may include a pixel arrayand ancillary processing circuits as shown in FIG. 2 and also describedbelow.

In one embodiment, the processor 19 may be a central processing unit(CPU), which can be a general-purpose microprocessor. As used herein,the terms “processor” and “CPU” may be used interchangeably for ease ofdescription. It is, however, understood that instead of or in additionto the CPU, the processor 19 may contain any other type of processorssuch as, a microcontroller, a digital signal processor (DSP), a graphicsprocessing unit (GPU), a dedicated application specific integratedcircuit (ASIC) processor, and the like. Furthermore, in one embodiment,the processor/host 19 may include more than one CPU, which may beoperative in a distributed processing environment. The processor 19 maybe configured to execute instructions and to process data according to aparticular instruction set architecture (ISA) such as, for example, anx86 instruction set architecture (32-bit or 64-bit versions), a PowerPC®ISA, or a MIPS (microprocessor without interlocked pipeline stages)instruction set architecture relying on RISC (reduced instruction setcomputer) ISA. In one embodiment, the processor 19 may be a system onchip (SoC) having functionalities in addition to a CPU functionality.

In particular embodiments, the memory module 20 may be a dynamic randomaccess memory (DRAM) such as, a synchronous DRAM (SDRAM), or aDRAM-based three dimensional stack (3DS) memory module such as, a highbandwidth memory (HBM) module, or a hybrid memory cube (HMC) memorymodule. In other embodiments, the memory module 20 may be a solid-statedrive (SSD), a non-3DS DRAM module, or any other semiconductor-basedstorage system such as, a static random access memory (SRAM), aphase-change random access memory (PRAM or PCRAM), a resistive randomaccess memory (RRAM or ReRAM), a conductive-bridging RAM (CBRAM), amagnetic RAM (MRAM), a spin-transfer torque MRAM (STT-MRAM), and thelike.

FIG. 2 depicts an example operational layout of the system 15 in FIG. 1according to the subject matter disclosed herein. The system 15 may beused to obtain depth information (along the Z-axis) for a 3D object,such as the 3D object 26, which may be an individual object or an objectwithin a scene (not shown). In one embodiment, the depth information maybe determined, or calculated, by the processor 19 based on the scan datareceived from the image sensor unit 24. In another embodiment, the depthinformation may be determined, or calculated, by the image sensor unit24 itself such as, for example, in case of the image sensor unit in theembodiment of FIG. 6A. In particular embodiments, the depth informationmay be used by the processor 19 as part of a 3D user interface to enablethe user of the system 15 to interact with the 3D image of the object oruse the 3D image of the object as part of games or other applicationsrunning on the system 15. The 3D imaging according to the subject matterdisclosed herein may be used for other purposes or applications as well,and may be applied to substantially any scene or 3D objects.

In FIG. 2, the X-axis is taken to be the horizontal direction along thefront of the device 15, the Y-axis is the vertical direction (out of thepage in this view), and the Z-axis extends away from the device 15 inthe general direction of the object 26 being imaged. For the depthmeasurements, the optical axes of the modules 22 and 24 may be parallelto the Z-axis. Other optical arrangements may be used as well toimplement the principles described herein, and these alternativearrangements are considered to be within the scope of the subject matterdisclosed herein.

The light source module 22 may illuminate the 3D object 26 as depictedby example arrows 28 and 29 associated with corresponding dotted lines30 and 31 representing an illumination path of a light beam or opticalradiation that may be used to point scan the 3D object 26 within anoptical field of view. A line-by-line point scan of the object surfacemay be performed using an optical radiation source, which, in oneembodiment, may be a laser light source 33 operated and controlled by alaser controller 34. A light beam from the laser source 33 may be pointscanned under the control of the laser controller 34 in the X-Ydirection across the surface of the 3D object 26 via projection optics35. The point scan may project light spots on the surface of the 3Dobject along a scan line, as described in more detail with reference toFIGS. 3 and 4 below. The projection optics may be a focusing lens, aglass/plastics surface, or other cylindrical optical element thatconcentrates laser beam from the laser 33 as a point or spot on thatsurface of the object 26. In the embodiment of FIG. 2, a convexstructure is depicted as a focusing lens 35. Any other suitable lensdesign may, however, be selected for projection optics 35. The object 26may be placed at a focusing location where illuminating light from thelight source 33 is focused by the projection optics 35 as a light spot.Thus, in the point scan, a point or narrow area/spot on the surface ofthe 3D object 26 may be illuminated sequentially by the focused lightbeam from the projection optics 35.

In particular embodiments, the light source (or illumination source) 33may be a diode laser or a light emitting diode (LED) emitting visiblelight, an NIR laser, a point light source, a monochromatic illuminationsource (such as, a combination of a white lamp and a monochromator) inthe visible light spectrum, or any other type of laser light source. Thelaser 33 may be fixed in one position within the housing of the device15, but may be rotatable in X-Y directions. The laser 33 may be X-Yaddressable (for example, by the laser controller 34) to perform pointscan of the 3D object 26. In one embodiment, the visible light may besubstantially green light. The visible light illumination from the lasersource 33 may be projected onto the surface of the 3D object 26 using amirror (not shown), or the point scan may be completely mirror-less. Inparticular embodiments, the light source module 22 may include more orless components than those depicted in the example embodiment of FIG. 2.

In the embodiment of FIG. 2, the light reflected from the point scan ofthe object 26 may travel along a collection path indicated by arrows 36and 37 and dotted lines 38 and 39. The light collection path may carryphotons reflected from or scattered by the surface of the object 26 uponreceiving illumination from the laser source 33. It is noted here thatthe depiction of various propagation paths using solid arrows and dottedlines in FIG. 2 (and also in FIGS. 3 and 4, as applicable) is forillustrative purpose only. The depiction should not be construed toillustrate any actual optical signal propagation paths. In practice, theillumination and collection signal paths may be different from thosedepicted in FIG. 2, and may not be as clearly-defined as in thedepiction in FIG. 2.

The light received from the illuminated object 26 may be focused ontoone or more pixels of a 2D pixel array 42 via collection optics 44 inthe image sensor unit 24. Like the projection optics 35, the collectionoptics 44 may be a focusing lens, a glass/plastics surface, or othercylindrical optical element that concentrates the reflected lightreceived from the object 26 onto one or more pixels in the 2D array 42.In the embodiment of FIG. 2, a convex structure is shown as a focusinglens 44. Any other suitable lens design may, however, be selected forcollection optics 44. Furthermore, for ease of illustration, only a 3×3pixel array is depicted in FIG. 2 (and also in FIG. 5). It isunderstood, however, that modern pixel arrays contain thousands or evenmillions of pixels. The pixel array 42 may be an RGB pixel array inwhich different pixels may collect light signals of different colors. Asmentioned before, in particular embodiments the pixel array 42 may beany 2D sensor such as, a 2D RGB sensor with IR cut filter, a 2D IRsensor, a 2D NIR sensor, a 2D RGBW sensor, a 2D RGB-IR sensor, and thelike. As described in more detail later, the system 15 may use the samepixel array 42 for 2D RGB color imaging of the object 26 (or a scenecontaining the object) as well as for 3D imaging (involving depthmeasurements) of the object 26. Additional architectural details of thepixel array 42 are described later with reference to FIG. 5.

The pixel array 42 may convert the received photons into correspondingelectrical signals that are then processed by the associated imageprocessing unit 46 to determine the 3D-depth image of the object 26. Inone embodiment, the image processing unit 46 may use triangulation fordepth measurements. The triangulation approach is described withreference to FIG. 3. The image processing unit 46 may also includerelevant circuits for controlling the operation of the pixel array 42.Exemplary image processing and control circuits are depicted in FIGS. 6Aand 6B.

The processor 19 may control the operations of the light source module22 and the image sensor unit 24. For example, the system 15 may have amode switch (not shown) controllable by the user to switch from2D-imaging mode to 3D-imaging mode. When the user selects the 2D-imagingmode using the mode switch, the processor 19 may activate the imagesensor unit 24, but may not activate the light source module 22 because2D imaging may use ambient light. On the other hand, when the userselects 3D imaging using the mode switch, the processor 19 may activateboth of the modules 22 and 24, and may also trigger change in the levelof the reset (RST) signal in the image processing unit 46 to switch froma linear mode to a logarithmic mode of imaging, for example, when theambient light is too strong to be rejected by linear mode (as describedbelow). The processed image data received from the image processing unit46 may be stored by the processor 19 in the memory 20. The processor 19may also display the user-selected 2D or 3D image on a display screen(not shown) of the device 15. The processor 19 may be programmed insoftware or firmware to carry out various processing tasks describedherein. Alternatively or additionally, the processor 19 may compriseprogrammable hardware logic circuits for carrying out some or all of itsfunctions. In particular embodiments, the memory 20 may store programcode, look-up tables (like the one showed in FIG. 8), and/or interimcomputational results to enable the processor 19 to carry out itsfunctions.

Briefly, the system 15 (more specifically, the processor 19) may performa one-dimensional (1D) point scan of a 3D object, such as the object 26in FIG. 2, along a scanning line using a light source, such as the lightsource module 22 in FIG. 2. As part of the point scan, the light sourcemodule 22 may be configured, for example, by the processor 19, toproject a sequence of light spots on a surface of the 3D object 26 in aline-by-line manner. The pixel processing unit 46 in the system 15 mayselect a row of pixels in an image sensor, such as the 2D pixel array 42in FIG. 2. The image sensor 42 has a plurality of pixels arranged in a2D array forming an image plane, and the selected row of pixels forms anepipolar line of the scanning line on the image plane. A briefdescription of epipolar geometry is provided in connection with FIG. 3.The pixel processing unit 46 may be operatively configured by theprocessor 19 to detect each light spot using a corresponding pixel inthe row of pixels. It is observed here that light reflected from anilluminated light spot may be detected by a single pixel or more thanone pixel such as, when the light reflected from the illuminated spotgets focused by the collection optics 44 onto two or more adjacentpixels. On the other hand, it may be possible that light reflected fromtwo or more light spots may be collected at a single pixel in the 2Darray 42. The timestamp-based approach described below removes depthcalculation-related ambiguities resulting from imaging of two differentspots by the same pixel or imaging of a single spot by two differentpixels. The image processing unit 46, as suitably configured by theprocessor 19, may generate a pixel-specific output in response to apixel-specific detection of a corresponding light spot in the sequenceof light spots. Consequently, the image processing unit 46 may determinethe 3D distance (or depth) to the corresponding light spot on thesurface of the 3D object based at least on the pixel-specific output andon a scan angle used by the light source for projecting thecorresponding light spot. The depth measurement is described in moredetail with reference to FIG. 3.

FIG. 3 is an example depiction of how a point scan may be performed for3D-depth measurements according to the subject matter disclosed herein.In FIG. 3, the X-Y rotational capabilities of the laser source 33 aredepicted using the arrows 62 and 64 depicting angular motion of thelaser in the X-direction (having angle “β”) and in the Y-direction(having angle “α”). In one embodiment, the laser controller 34 maycontrol the X-Y rotation of the laser source 33 based on scanninginstructions/input received from the processor 19. For example, if theuser selects 3D-imaging mode, the processor 19 may instruct the lasercontroller 34 to initiate 3D-depth measurements of the object surfacefacing the projection optics 35. In response, the laser controller 34may initiate a 1D X-Y point scan of the object surface through X-Ymovement of the laser light source 33. As depicted in FIG. 3, the laser33 may point scan the surface of the object 26 by projecting light spotsalong 1D horizontal scanning lines, two of which S_(R) 66 and S_(R+1) 68are identified by dotted lines in FIG. 3. Because of the curvature ofthe surface of the object 26, the light spots 70-73 may form thescanning line S_(R) 66 in FIG. 3. For ease of description and clarity,the light spots constituting the scan line S_(R+1) 68 are not identifiedusing reference numerals. The laser 33 may scan the object 26 along rowsR, R+1, and so on, one spot at a time, for example, in the left-to-rightdirection. The values of “R,” “R+1,” and so on, are with reference torows of pixels in the 2D pixel array 42 and, hence, these values areknown. For example, in the 2D pixel array 42 in FIG. 3, the pixel row“R” is identified using reference numeral “75” and the row “R+1” isidentified using reference numeral “76.” It is understood that rows “R”and “R+1” are selected from the plurality of rows of pixels forillustrative purpose only.

The plane containing the rows of pixels in the 2D pixel array 42 may becalled the image plane, whereas the plane containing the scanning lines,like the lines S_(R) and S_(R+1), may be called the scanning plane. Inthe embodiment of FIG. 3, the image plane and the scanning plane areoriented using epipolar geometry such that each row of pixels R, R+1,and so on, in the 2D pixel array 42 forms an epipolar line of thecorresponding scanning line S_(R), S_(R+1), and so on. A row of pixels“R” may be considered epipolar to a corresponding scanning line “S_(R)”when a projection of an illuminated spot (in the scanning line) onto theimage plane may form a distinct spot along a line that is the row “R”itself. For example, in FIG. 3, the arrow 78 illustrates theillumination of the light spot 71 by the laser 33, whereas the arrow 80shows that the light spot 71 is being imaged or projected along the row“R” 75 by the focusing lens 44. Although not depicted in FIG. 3, it isobserved that all of the light spots 70-73 will be imaged bycorresponding pixels in the row “R.” Thus, in one embodiment, thephysical arrangement, such as the position and orientation, of the laser33 and the pixel array 42 may be such that illuminated light spots in ascanning line on the surface of the object 26 may be captured ordetected by pixels in a corresponding row in the pixel array 42 and thatrow of pixels thus forms an epipolar line of the scanning line.

It is understood that the pixels in the 2D pixel array 42 may bearranged in rows and columns. An illuminated light spot may bereferenced by its corresponding row and column in the pixel array 42.For example, in FIG. 3, the light spot 71 in the scanning line S_(R) isdesignated as “X_(R,i)” to indicate that the spot 71 may be imaged byrow “R” and column “i” (C_(i)) in the pixel array 42. The column C_(i)is indicated by dotted line 82. Other illuminated spots may be similarlyidentified. As noted before, it may be possible that light reflectedfrom two or more lights spots may be received by a single pixel in arow, or, alternatively, light reflected from a single light spot may bereceived by more than one pixel in a row of pixels. The timestamp-basedapproach described later may remove the ambiguities in depthcalculations arising from such multiple or overlapping projections.

In the depiction of FIG. 3, the arrow having reference numeral “84”represents the depth or distance “Z” (along the Z-axis) of the lightspot 71 from the X-axis along the front of the device 15, such as theX-axis depicted in FIG. 2. In FIG. 3, a dotted line having the referencenumeral “86” represents such axis, which may be visualized as beingcontained in a vertical plane that also contains the projection optics35 and the collection optics 44. For ease of explanation of thetriangulation method, however, the laser source 33 is depicted in FIG. 3as being on the X-axis 86 instead of the projection optics 35. In atriangulation-based approach, the value of Z may be determined using thefollowing equation:

$\begin{matrix}{Z = {\frac{hd}{q - {h\mspace{14mu} \tan \mspace{14mu} \theta}}.}} & (1)\end{matrix}$

The parameters used in Eq. (1) are also indicated in FIG. 3. Based onthe physical configuration of the device 15, the values for theparameters on the right side of Eq. (1) may be pre-determined. In Eq.(1), the parameter h is the distance (along the Z-axis) between thecollection optics 44 and the image sensor (which is assumed to be in avertical plane behind the collection optics 44); the parameter d is theoffset distance between the light source 33 and the collection optics 44associated with the image sensor 24; the parameter q is the offsetdistance between the collection optics 44 and a pixel that detects thecorresponding light spot, in this case the detecting/imaging pixel i isrepresented by column C_(i) associated with the light spot X_(R,i) 71;and the parameter θ is the scan angle or beam angle of the light sourcefor the light spot under consideration, in this case the light spot 71.Alternatively, the parameter q may also be considered as the offset ofthe light spot within the field of view of the pixel array 42.

It is seen from Eq. (1) that only the parameters θ and q are variablefor a given point scan; the other parameters h and d are essentiallyfixed due to the physical geometry of the device 15. Because the row R75 is an epipolar line of the scanning line S_(R), the depth differenceor depth profile of the object 26 may be reflected by the image shift inthe horizontal direction, as represented by the values of the parameterq for different lights spots being imaged. As described below, thetime-stamp based approach according to particular embodiments disclosedherein may be used to find the correspondence between the pixel locationof a captured light spot and the corresponding scan angle of the lasersource 33. In other words, a timestamp may represent an associationbetween the values of parameters q and θ. Thus, from the known value ofthe scan angle θ and the corresponding location of the imaged light spot(as represented by the parameter q), the distance to that light spot maybe determined using the triangulation Eq. (1).

It is observed here that usage of triangulation for distancemeasurements is described in the relevant literature including, forexample, the United States Patent Published Patent Application No.2011/0102763 to Brown et al. The disclosure in the Brown publicationrelating to triangulation-based distance measurement is incorporatedherein by reference in its entirety.

FIG. 4 depicts an exemplary time-stamping for scanned light spotsaccording to the subject matter disclosed herein. Additional details ofgeneration of individual timestamps are provided below with reference toFIG. 7. In contrast to FIG. 3, in the embodiment of FIG. 4, thecollection optics 44 and the laser 33 are depicted in an offsetarrangement to reflect the actual physical geometry of these componentsas depicted in the embodiment of FIG. 2. By way of an example, thescanning line 66 is shown in FIG. 4 along with corresponding light spots70-73, which, as mentioned before, may be projected based on aleft-to-right point scan of the object surface by the sparse laser pointsource 33. Thus, as depicted, the first light spot 70 may be projectedat time instant t₁, the second light spot 71 may be projected at timeinstant t₂, and so on. These light spots may be detected/imaged byrespective pixels 90-93 in the pixel row R 75, which is an epipolar lineof the scanning line S_(R) as described earlier. In one embodiment, thecharge collected by each pixel when detecting a light spot may be in theform of an analog voltage, which may be output to the image processingunit 46 for pixel-specific depth determination as described below. Theanalog pixel outputs (pixouts) are collectively indicated by arrow 95 inFIG. 4.

As shown in FIG. 4, each detecting pixel 90-93 in row R may have anassociated column number, here, columns C₁ through C₄. Furthermore, itis seen from FIG. 3 that each pixel column C_(i) (i=1, 2, 3, and so on)has an associated value for the parameter q in Eq. (1). Thus, when apixel-specific timestamp t₁-t₄ is generated for the detecting pixels90-93 (as described in more detail below), the timestamp may provide anindication of the column number of the pixel and, hence, thepixel-specific value of the parameter q. Additionally, in oneembodiment, the spot-by-spot detection using pixels in the pixel array42 may allow the image processing unit 46 to “link” each timestamp withthe corresponding illuminated spot and, hence, with the spot-specificscan angle θ because the laser 33 may be suitably controlled toilluminate each spot in the desired sequence with pre-determined valuesfor spot-specific scan angles θ. Thus, timestamps provide correspondencebetween the pixel location of a captured laser spot and its respectivescan angle in the form of the values for parameters q and θ in Eq. (1)for each pixel-specific signal received from the pixel array 42. Aspreviously described, the values of the scan angle and the correspondinglocation of the detected spot in the pixel array 42, as reflectedthrough the value of the parameter q in Eq. (1), may allow depthdetermination for that light spot. In this manner, the 3D depth map forthe surface of the object 26 in the field of view of the pixel array 42may be generated.

FIG. 5 depicts example circuit details of the 2D pixel array 42 and aportion of the associated processing circuits in the image processingunit 46 of the image sensor 24 in FIGS. 1 and 2 according to the subjectmatter disclosed herein. As previously noted, the pixel array 42 isdepicted having nine pixels 100-108 arranged as a 3×3 array for ease ofdescription only. In practice, a pixel array may contain hundreds ofthousands or millions of pixels in multiple rows and columns. In oneembodiment, each pixel 100-108 may have an identical configuration asdepicted in FIG. 5. In the embodiment of FIG. 5, the 2D pixel array 42may be a complementary metal oxide semiconductor (CMOS) array in whicheach pixel is a four transistor pinned photo-diode (4T PPD) pixel. Forease of description, the constituent circuit elements of only pixel 108are labeled with reference numerals. The following description of theoperation of the pixel 108 equally applies to the other pixels 101-107and, hence, the operation of each individual pixel is not describedherein.

As depicted, the 4T PPD pixel 108 (and similar other pixels 101-107) mayform a pinned photo-diode (PPD) 110 and four n-channel metal oxidesemiconductor field effect transistors (NMOS) 111-114 connected asdepicted. The transistor 111 may operate as a transfer gate (TG),floating diffusion (FD) transistor. Generally, the 4T PPD pixel 108 mayoperate as follows: The PPD 110 may first convert incident photons intoelectrons, thereby converting an optical input signal into an electricalsignal in the charge domain. Then, the transfer gate 111 may be “closed”to transfer all the photon-generated electrons from the PPD 110 to thefloating diffusion. The signal in the charge domain thus is converted tothe voltage domain for convenience of subsequent processing andmeasurements. The voltage at the floating diffusion may be latertransferred as a pixout signal to an analog-to-digital converter (ADC)using the transistor 114 and converted into an appropriate digitalsignal for subsequent processing. More details of the pixel output(PIXOUT) generation and processing are provided below with reference toFIGS. 7, 10 and 11.

In the embodiment of FIG. 5, a row decoder/driver 116 in the imageprocessing unit 46 is depicted to provide three different signals tocontrol the operation of the pixels in the pixel array 42 to generatethe column-specific pixout signals 117-119. In the embodiment of FIG. 4,the output 95 may collectively represent such PIXOUT signals 117-119. Arow select (RSEL) signal may be asserted to select an appropriate row ofpixels. In one embodiment, the row to be selected is the epipolar lineof the current scanning line (of light spots) being projected by thelaser source 33. The row decoder/driver 116 may receive the address orcontrol information for the row to be selected via the rowaddress/control inputs 126 from, for example, the processor 19. In thepresent description, it is assumed that the row decoder/driver 116selects the row of pixels containing the pixel 108. A transistor, suchas the transistor 114, in each row of pixels in the pixel array 42 maybe connected to a respective RSEL line 122-124 as depicted. A reset(RST) signal may be applied to pixels in the selected row to reset thepixels of the row to a predetermined high voltage level. Eachrow-specific RST signal 128-130 is shown in FIG. 5 and explained in moredetail in connection with the waveforms in FIGS. 7, 10 and 11. Atransistor, such as the transistor 112, in each pixel may receive therespective RST signal as depicted. A transfer (TX) signal may beasserted to initiate transfer of the pixel-specific output voltage(PIXOUT) for subsequent processing. Each row-specific TX line 132-134 isshown in FIG. 5. A transfer-gate transistor, such as the transistor 111,may receive the respective TX signal as depicted in FIG. 5.

As previously mentioned, in particular embodiments disclosed herein, the2D array 42 and the rest of the rest of the components in the imagesensor unit 24 may be used for 2D RGB (or non-RGB) imaging as well asfor 3D depth measurements. Consequently, as depicted in FIG. 5, theimage sensor unit 24 may include a pixel column unit 138 that includescircuits for correlated double sampling (CDS) as well as column-specificADCs (one ADC per column of pixels) to be used during 2D and 3D imaging.The pixel column unit 138 may receive the PIXOUT signals 117-119 andprocess the PIXOUT signals to generate a digital data output (Dout)signal 140 from which 2D image may be generated or 3D-depth measurementscan be obtained. The pixel column unit 138 may also receive a referenceinput 142 and a ramp input 143 during processing of the PIXOUT signals117-119. More details of the operation of the unit 138 are providedlater below. In the embodiment of FIG. 5, a column decoder unit 145 isdepicted coupled to the pixel column unit 138. The column decoder 145may receive a column address/control input 147 from, for example, theprocessor 19, for the column to be selected in conjunction with a givenrow select (RSEL) signal. The column selection may be sequential,thereby allowing sequential reception of the pixel output from eachpixel in the row selected by the corresponding RSEL signal. Theprocessor 19 may be aware of the currently-projected scanning line oflight spots and, hence, may provide appropriate row address inputs toselect the row of pixels that forms the epipolar line of the currentscanning line and may also provide appropriate column address inputs toenable the pixel column unit 138 to receive outputs from the individualpixels in the selected row.

It is observed here that although the description herein primarilyfocuses on the 4T PPD pixel design shown in FIG. 5 for 2D and 3D imagingaccording to the subject matter disclosed herein, different types ofpixels may be used in the pixel array 42 in other embodiments. Forexample, in one embodiment, each pixel in the pixel array 42 may be a 3Tpixel, which omits the transfer gate transistor, like the transistor 111in the 4T PPD design in FIG. 5. In other embodiments, 1T pixels or 2Tpixels may be used as well. In yet another embodiment, each pixel in thepixel array 42 may have a shared-transistor pixel configuration in whichtransistors and read-out circuitry may be shared among two or moreneighboring pixels. In the shared-transistor pixel configuration, eachpixel may have at least one photo-diode and one transfer-gatetransistor; the rest of the transistors may be shared among two or morepixels. One example of such a shared-transistor pixel is the 2-shared(1×2) 2.5T pixel in which five transistors (T) are used for two pixels,resulting in a 2.5T/pixel configuration. Another example of ashared-transistor pixel that may be used in the pixel array 42 is the1×4 4-shared pixel, in which 4 pixels share the readout circuitry, buteach one has at least one photo-diode and one TX (transfer-gate)transistor. Other pixel configurations than those listed here may besuitably implemented for 2D and 3D imaging as the subject matterdisclosed herein.

FIG. 6A depicts an exemplary layout of an image sensor unit, such as theimage sensor unit 24 in FIG. 5, according to one embodiment of thesubject matter disclosed herein. For the sake of brevity, only a briefdescription of the architecture in FIG. 6A is provided herein; morerelevant operational details are provided later in connection with FIGS.7, 10 and 11. As depicted, the image sensor unit 24 in FIG. 6A mayinclude a row decoder unit 149 and a row driver unit 150, both of whichmay collectively form the row decoder/driver 116 in FIG. 5. Although notshown in FIG. 6A, the row decoder unit 149 may receive a row addressinput (like the input 126 depicted in FIG. 5) from, for example, theprocessor 19, and may decode the input to enable the row driver unit 150to provide appropriate RSEL, RST, and TX signals to the rowselected/decoded by the row decoder 149. The row driver unit 150 mayalso receive control signals (not shown) from, for example, theprocessor 19, to configure the row driver 150 to apply appropriatevoltage levels for the RSEL, RST, and TX signals. In the image sensor 24in FIG. 6A, a column ADC unit 153 may represent the pixel column unit138 in FIG. 5. For ease of depiction, in FIG. 6A, various row-specificdriver signals, such as the RSEL, RST, and TX signals, from the rowdriver 150 are collectively referenced using a single reference numeral155. Similarly, all column-specific pixel outputs (pixouts), like thepixouts 117-119 in FIG. 5, are collectively referenced using a singlereference numeral 157. The column ADC unit 153 may receive the pixoutsignals 157, the reference input 142 (from a reference signal generator159) and the ramp signal 143 to generate a pixel-specific output by thecorresponding column-specific ADC for the column of a pixel. The 2Dimaging is described in more detail in connection with reference to FIG.10. In one embodiment, the ADC unit 153 may include circuitry for CDS,as in the case of the pixel column unit 138 in FIG. 5, to generate a CDSoutput (not shown) that is the difference between the reset level of thepixel and the received signal level. In particular embodiments, the3D-depth values may be combined with the 2D image to generate a 3D imageof the object.

The column ADC unit 153 may include a separate ADC per pixel column inthe 2D array 42. Each column-specific ADC may receive a respective rampinput 143 (from a ramp signal generator 163) along with the pixoutsignals 157. In one embodiment, the ramp signal generator 163 maygenerate the ramp input 143 based on the reference voltage levelreceived from the reference signal generator 159. Each column-specificADC in the ADC unit 153 may process the received inputs to generate thecorresponding digital data output (Dout) signal 140. From the columndecoder 145, the ADC unit 153 may receive information about which columnADC output to be readout and sent to the Dout bus 140, and may alsoreceive information about which column to select for a given row toreceive the appropriate pixel output. Although not depicted in FIG. 6A,the column decoder unit 145 may receive a column address input (like theinput 147 in FIG. 5), for example, from the processor 19, and decode theinput to enable the column ADC unit 153 to select the appropriate pixelcolumn. In the embodiment of FIG. 6A, the decoded column address signalsare collectively identified using the reference numeral 165.

The digital data outputs 140 from the ADC units may be processed by adigital processing block 167. In one embodiment, for the 2D RGB imagingmode, each ADC-specific data output 140 may be a multi-bit digital valuethat substantially corresponds to the actual photon charge collected bythe respective pixel. On the other hand, in the 3D-depth measurementmode, each ADC-specific data output 140 may be a timestamp valuerepresenting the time instant when the respective pixel detects itscorresponding light spot. This timestamping approach according to theteachings of the present disclosure is described later in more detail.The digital processing block 167 may include circuits to provide timinggeneration; image signal processing (ISP) such as, processing of dataoutputs 140 for the 2D-imaging mode; depth calculations for the3D-imaging mode; and so on. In that regard, the digital processing block167 may be coupled to an interface unit 168 to provide the processeddata as an output 170, for example, to enable the processor 19 to rendera 2D RGB/non-RGB image or a 3D depth image of the 3D object 26 on adisplay screen (not shown) of the device 15. The interface unit 168 mayinclude a phase-locked loop (PLL) unit for generation of clock signalsthat support the timing generation functionality in the digitalprocessing block 167. Furthermore, the interface unit 168 may alsoinclude a mobile industry processor interface (MIPI) that provides anindustry-standard hardware and software interface to other components orcircuit elements in the device 15 for the data generated by the digitalblock 167. The MIPI specifications support a broad range of mobileproducts and provide specifications for a camera of a mobile device,display screen, power management, battery interface, and the like. TheMIPI-standardized interfaces may yield an improved operability betweenperipherals of a mobile device, such as a camera or a display screen ofa smartphone, and the application processor(s) of the mobile device,which may not be from the same vendor as the vendor (or vendors)providing the peripherals.

In the embodiment of FIG. 6A, a timestamp calibration unit 171 isdepicted coupled to the column ADC unit 153 to provide appropriatecalibration signals 172 to individual column-specific ADCs to enableeach column-specific ADC unit to generate an output representing apixel-specific timestamp value in the 3D-measurement mode. Thistimestamping approach is described in more detail in connection withFIG. 7.

FIG. 6B depicts architectural details of an example CDS+ADC unit 175 for3D-depth measurement according to one embodiment of the subject matterdisclosed herein. For ease of description, the unit 175 may be referredbelow to as an ADC unit, however, it is understood that the unit 175 mayalso include CDS functionality in addition to the ADC functionality. Asimplified version of a CDS unit is represented using the capacitor 176in FIG. 6B. In one embodiment, each column of pixels in the 2D pixelarray 42 may have a column-specific, single-slope ADC unit similar tothe ADC unit 175. Thus, in the embodiment of FIG. 5, there may be threeADC units in the pixel column unit 138, one ADC per column. As depicted,the ADC 175 in the embodiment of FIG. 6B may include two operationaltransconductance amplifiers (OTA) 177 and 179 connected in series with abinary counter 181 and a line memory unit 183. For ease of description,only the inverting (−) and non-inverting (+) voltage inputs to the OTAs177 and 179 are depicted in FIG. 6B; the biasing inputs and the powersupply connections are not shown. It is understood that an OTA is anamplifier in which a differential input voltage produces an outputcurrent. Thus, an OTA may be considered as a voltage-controlled currentsource. The biasing inputs may be used to provide currents or voltagesto control the transconductance of the amplifier. The first OTA 177 mayreceive from the CDS unit 176 a CDS version of the pixout voltage from apixel, such as the pixel 108 in FIG. 5 that is selected in the activatedrow using the column number received from the column decoder 145. TheCDS version of a pixout signal may be referred to as a “PIX_CDS” signal.The OTA 177 may also receive a Vramp voltage 143 from the ramp signalgenerator 163 (FIG. 6A). The OTA 177 may generate an output current whenthe pixout voltage 157 drops below the Vramp voltage 143, as describedin connection with FIG. 7. The output of the OTA 177 may be filtered bythe second OTA 179 before being applied to the binary counter 181. Inone embodiment, the binary counter 181 may be a 10-bit ripple counterthat receives a clock (Clk) input 185 and generates a timestamp value186 based on the clock cycles counted during a pre-determined timetriggered by the generation of the output current by the first OTA 177.In the context of the embodiment in FIG. 6A, the Clk input 185 may be asystem-wide clock or an image sensor-specific clock generated by the PLLunit 168 or other clock generator (not shown) in the device 15. Thepixel-specific timestamp value 186 may be stored in the line memory 183against the column number (column #) of the pixel, and subsequentlyoutput to the digital processing block 167 as the Dout signal 140. Thecolumn number input 165 may be received from the column decoder unit 145depicted in FIG. 6A.

In particular embodiments, the RGB color model may be used for sensing,representation, and display of images on mobile devices such as, forexample, the device 15 in FIGS. 1 and 2. In the RGB color model, thelight signals having three primary colors—red, green, and blue—may beadded together in various ways to produce a broad array of colors in thefinal image. The CDS method may be used in 2D RGB imaging to measure anelectrical value, such as a pixel/sensor output voltage, in a mannerthat allows removal of an undesired offset. For example, a CDS unit,like the CDS unit 176, may be employed in each column-specific ADC unit,like the ADC unit 175, to perform CDS. In CDS, the output of the pixelmay be measured twice—once in a known condition, and once in an unknowncondition. The value measured from the known condition may be thensubtracted, or removed, from the value measured from the unknowncondition to generate a value with a known relation to the physicalquantity being measured—here, the photoelectron charge representing thepixel-specific portion of the image signal. Using CDS, noise may bereduced by removing the reference voltage of the pixel (such as, forexample, the voltage of a pixel after being reset) from the signalvoltage of the pixel at the end of each integration period. Thus, inCDS, before the charge of a pixel is transferred as an output, the resetvalue is sampled. The reference value is deducted, or removed, from thevalue after the charge of the pixel is transferred.

It is observed here that, in particular embodiments, the ADC unit 175may be used for both 2D imaging as well as 3D-depth measurements. Allthe inputs for such shared configuration, however, are not depicted inFIG. 6B. In the shared use case, the corresponding Vramp signal may bedifferent as well for 2D imaging.

FIG. 7 depicts a timing diagram 190 that shows example timing ofdifferent signals in the system 15 of FIGS. 1 and 2 to generatetimestamp-based pixel-specific outputs in a 3D-linear mode of operationaccording to particular embodiments of the subject matter disclosedherein. As noted before, in particular embodiments, all pixels in thesame image sensor 24 may be used for 2D as well as 3D imaging. The3D-depth measurements may, however, be performed using a 3D-linear modeor a 3D-logarithmic mode depending on the level of ambient light. Asdescribed in more detail in connection with FIG. 11, the 3D-logarithmicmode may be used for depth measurements when ambient light rejection isneeded. The description of FIG. 7, however, relates to the timingwaveforms associated with the 3D-linear mode.

Briefly, as described earlier in connection with FIGS. 3 and 4, the 3Dobject 26 may be point-scanned, one spot at a time, by the laser lightsource 33 along a row R 75 of the pixel array 42 in which R is knownbased on its epipolar relation with the scanning line S_(R) 66. Afterscanning one row, the scanning operation repeats with another row. Whenthe laser projects the next spot, the earlier-projected light spot maybe imaged by the corresponding pixel in the row R. The pixel-specificoutputs from all the pixels in the row R may be read out to the depthprocessing circuit/module in the digital processing block 167 (FIG. 6A).

To generate a pixel-specific output, the corresponding row may have tobe initially selected using an RSEL signal. In the context of FIG. 7, itis assumed that the row decoder/driver 116 in FIG. 5 selects the row ofpixels containing pixels 106-108 by asserting the RSEL signal 122 to a“high” level as depicted in FIG. 7. Thus, all the pixels 106-108 areselected together. For ease of description, the same reference numeralsare used in FIG. 7 for the signals, inputs, or outputs that are alsodepicted in FIGS. 5 and 6. Initially, all the pixels 106-108 in theselected row may be reset to a high voltage using the RST line 128. The“reset” level of a pixel may represent an absence of the pixel-specificdetection of a corresponding light spot. In the 3D-linear mode accordingto one embodiment of the present disclosure, the RST signal 128 may bereleased from its high level for a pre-determined time to facilitateintegration of photoelectrons received by the pixels 106-108 to obtainthe corresponding pixel output (pixout) signals 117-119, two of whichare depicted in FIG. 7 and described below. The PIXOUT1 signal 119represents the output supplied to a corresponding ADC unit by the pixel108, and is shown using a dashed line having the pattern “- ⋅⋅ - ⋅⋅ -.”The PIXOUT2 signal 118 represents the output supplied to a correspondingADC unit by the pixel 107, and is shown using a dashed line having thepattern “⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅.” On the other hand, in the 3D logarithmic modeaccording to one embodiment disclosed herein, the RST signal may remainhigh for the selected row during generation of the pixel output asdescribed below. It is noted here that, in one embodiment, other RSTlines, like the lines 129-130 in FIG. 5, may remain high or “on” forunselected rows to prevent blooming. It is noted here that, strictlyspeaking, the PIXOUT signals 118-119 in FIG. 7 (and similar pixoutsignals in FIGS. 10 and 11) may be slightly modified by a CDS unit suchas, the CDS unit 176 in FIG. 6B before being applied as PIX_CDS signalsto the first OTA like the OTA 177 in FIG. 6B—in a respectivecolumn-specific ADC unit, such as the ADC unit 175 in FIG. 6B. Forsimplicity of depiction, and ease of described, however, the PIXOUTsignals in FIGS. 7, 10 and 11 are treated as representatives ofrespective PIX_CDS signals (not shown) and are considered as having beendirectly “input” to the respective OTAs 177.

After reset, when a photodiode in a pixel receives incident luminance,such as, the photoelectrons in the light reflected from a light spotprojected on the surface of the 3D object 26, the photodiode maygenerate corresponding photocurrent. Detection of incident light by apixel may be called an “ON event,” whereas a decrease in the intensityof incident light may produce an “OFF event.” The photocurrent generatedin response to an ON event may decrease the pixel output voltage(PIXOUT) from its initial reset level. A pixel thus functions as atransducer to convert received luminance/light signal into acorresponding electrical (analog) voltage, which is generally designatedas a PIXOUT signal in FIGS. 5-7, 10 and 11. Each pixel may be readindividually and, preferably, in the sequence in which the correspondinglight spots are projected by the laser source. The analog pixout signalmay be converted to a digital value by the corresponding column ADC. Inthe 2D-imaging mode, the ADC may function as an analog-to-digitalconverter and generate a multi-bit output. In the 3D depth measurementmode, however, the ADC may function as a time-to-digital converter andgenerate a timestamp value representing the time when a light spot isdetected by a pixel.

Referring again to FIG. 7, after the pixel reset is done (with RST 128high), the column ADCs associated with pixels 106-108 may be reset aswell before the RST is released. The transfer (TX) signal 132 may,however, remain high throughout. The ADCs may be reset using either acommon ADC reset signal or individual ADC-specific reset signals. In theembodiment of FIG. 7, a common ADC_RST signal 192 is depicted to havebeen briefly asserted (to a high level) to reset the column-specificADCs, like the ADC 175, in the column ADC unit 153 (FIG. 6A). In oneembodiment, the ADCs may be reset to a pre-determined binary value, suchas a binary 0 or other known number, after the pixels are reset. In FIG.7, the reset values for ADCs associated with pixels 108 and 107 aredepicted by fields 194-195 in the signals ADCOUT1 (or ADC output “A”)and ADCOUT2 (or ADC output “B”), respectively. It is noted here that theterm “field” is used here for the sake of convenience only whendescribing the ADC outputs shown in FIGS. 7, 10 and 11. It is understoodthat an ADC output may not actually include all of such fields at thesame time, but may be a specific digital value depending on the currentstage of signal processing of the ADC. If the ADC is reset, its outputmay be a binary 0. If the ADC is triggered to count clock pulses, itsoutput may be a count value as in case of the 3D depth measurements inFIGS. 7 and 11. If the ADC is used for 2D color imaging (as in case ofFIG. 10), then its output may be a multi-bit value representing an imagesignal. Thus, the ADC output signals in FIGS. 7, 10 and 11 are depictedwith such “fields” merely to depict different digital values an ADC mayinternally generate in progressing toward the final output. In FIG. 7,the reference numeral 197 is used to refer to the ADCOUT1 signalrepresenting the output of the ADC associated with the pixel 108, andthe reference numeral 198 is used to refer to the ADCOUT2 signalrepresenting the output of the ADC associated with the pixel 107. Eachof the outputs 197-198 may appear as the Dout signal 140 (FIGS. 5 and 6)when the respective ADC is selected by the column decoder during memoryreadout. Prior to being reset, the ADC outputs 197-198 may have unknownvalues, as indicated by the notation “x” in the fields 199-200.

After ADCs are reset, a pre-determined threshold value may be enabled byde-asserting the ramp input (Vramp) 143 to a pre-defined voltage levelafter the pixel reset signal 128 and ADC reset signal 192 are released.In the embodiment of FIG. 7, the RAMP input 143 is common to allcolumn-specific ADCs, thereby providing the same Vramp voltage to eachADC. In other embodiments, however, different Vramp values may beapplied to two or more ADCs through separate, ADC-specific ramp inputs.Furthermore, in particular embodiments, the Vramp threshold may be aprogrammable parameter, allowing it to be variable as desired. After thethreshold (RAMP signal) is enabled, the pixel-specific ADCs may wait forthe corresponding “ON event” for the pixel before starting their binarycounters, like the counter 181 in FIG. 6B.

In the 3D-depth measurement mode, each ADC may generate a single bitoutput (representing binary 0 or 1), as opposed to a multi-bit output incase of the 2D-imaging mode (described below). Thus, in case of an RGBsensor, any color information received by a pixel in the RGB pixel array42 may be effectively ignored. In the absence of any incident lightdetected by a pixel, the corresponding ADCOUT signal may remain at thebinary 0 value. Thus, columns without any ON events may continue to havedigital value 0 (or other known number) for their respective ADCOUTsignals. As noted before, however, when a pixel receives incident light,its PIXOUT line may start to droop from its reset level, as indicated bythe downward slopes of the PIXOUT1 and PIXOUT2 signals in FIG. 7.Assuming that pixel charge is read starting with the pixel that receivesthe charge first, such a reading may start with the right-most pixel ina row and end with the left-most pixel as depicted in, for example, FIG.4 in which t₁ is the earliest time instant and t₄ is the latest one.Thus, in the embodiment of FIG. 7, the output of the pixel 108 (PIXOUT1)may be read before that of the pixel 107 (PIXOUT2). As soon as theprogressively-drooping PIXOUT1 reaches the Vramp threshold 143, thesingle-bit ADCOUT1 may flip from binary 0 to binary 1. Instead ofoutputting the bit 1, however, the corresponding ADC may record the timewhen the bit flips (from 0 to 1). In other words, the ADC associatedwith the pixel 108 may function as a time-to-digital converter, bystarting the binary counter in the ADC, as indicated by the “up count”field 202 in ADCOUT1. During the “up count” period, the counter in theADC may count the clock pulses in the CLK signal 185, which may beapplied to each ADC as depicted in, for example, FIG. 6B. The countedclock pulses are shown by the Counter Clock-1 signal 204 in FIG. 7, andthe counted value (in the “up count” field) may be provided as apixel-specific output for the pixel 108. A similar counting may occur atthe ADC associated with pixel 107 for the charge collected by the pixel107, as indicated by the Counter Clock-2 signal 205 in FIG. 7. Thepixel-specific counted value (in the “up count” field 207) may beprovided by the respective ADC as a pixel-specific output for the pixel107. After scanning all pixels in one row, the pixel-by-pixel chargecollection operation may repeat with another row, while the outputs fromthe earlier-scanned row are read out to the depth calculation unit inthe digital block 167.

Each ADC output may effectively represent a respective timestamp valueproviding a temporal indication of a detection by a pixel of a lightspot on the object surface illuminated by the laser light source 33. Atimestamp may be considered to capture the light arrival time for apixel. In one embodiment, a timestamp value may be generated for adetected light spot by the digital processing block 167 from the countvalue (of the counted clock pulses) received from an ADC unit. Forexample, the digital block 167 may generate a timestamp by relating thecount value to an internal system time or other reference time. Thetimestamp is generated at the receiving end and, hence, may notnecessarily represent the exact time when the corresponding light spotwas projected by the light source. The timestamp values may, however,allow the digital block 167 to establish a temporal correlation amongtime-stamped light spots, thereby allowing the digital block 167 todetermine distances to time-stamped light spots in the time-wise orderspecified by the temporal correlation, i.e., the distance to theearliest illuminated light spot being determined first, and so on, untilthe distance to the last-illuminated light spot is determined. In oneembodiment, the timestamping approach may also facilitate resolution ofthe ambiguity that may arise from multiple light spots being imaged onthe same pixel.

All ADC-based counters may stop simultaneously such as, for example,when the ramp signal 143 is asserted again after a pre-determined timeperiod has elapsed. In FIG. 7, the transition of the ramp signal 143,marking the conclusion of the pre-determined time period for pixelcharge integration, is indicated by dotted line 210. The RSEL 122 andthe RST 128 signals may also transition their states substantiallysimultaneously with the change in the level of the ramp signal 143 (atline 210). It is observed here that, in one embodiment, all ADC-basedcounters may be reset at line 210. In another embodiment, all ADC-basedcounters may be reset at any time prior to the selection of the next rowof pixels for reading the pixel charge. Despite resetting of ADCcounters upon conclusion of scanning of pixels in one row, the timestampvalue for each pixel in the pixel array 42 may remain distinct becauseof the relational establishment of the timestamp value against aninternal system time or other reference source of time, which may remainglobal and continuously-running.

It is observed here that, in the embodiment of FIG. 7, a later-scannedpixel, such as the pixel 107, may have a smaller ADC output than thepixel that is scanned earlier, such as the pixel 108. Thus, as depicted,the ADCOUT2 may have less count value (or less number of clock pulsescounted) than the ADCOUT1. Alternatively, in another embodiment, alater-scanned pixel may have a larger ADC output than an earlier-scannedpixel, for example, if each ADC-specific counter starts counting when apixel is reset and stops counting when an “ON event” is detected, suchas, when the pixout signal of the pixel droops below a given threshold(Vramp).

It is noted here that circuits and waveforms shown in FIGS. 5-7, 10 and11 are based on single-slope ADCs with per column up-counters. It isunderstood that the time-stamping approach may, however, be implementedwith up- or down-counters depending on the design choice. Furthermore,single-slope ADCs with global counters may be used as well. For example,in one embodiment, instead of using individual, column-based counters, aglobal counter (not shown) may be shared by all column ADCs. In thatcase, the ADCs may be configured such that the column memory, like theline memory 183 in FIG. 6B, in each ADC may latch the output of theglobal counter to generate an appropriate ADC-specific output when acolumn-based comparator unit (not shown) detects an “ON event” such as,when it first senses the respective pixout signal drooping below theramp threshold 143.

Although not depicted in FIG. 7, it is observed here that dark currentoffset can be removed by decreasing the Vramp threshold at a rate thatis the same as that of the dark current. Dark current may be arelatively small electric current that flows through a photosensitivedevice, such as a photodiode, even when no photons are entering thedevice. In image sensors, dark current may cause noise or unwantedartefacts in the collected charge. Dark current may be caused by defectsin pixels and may have an effect like the photocurrent. Thus, due to thedark current, the pixel output may still decrease even without theexistence of light (or in the absence of the light being received by thepixel). Thus, during charge collection, when the pixels in a row arescanned from right to left, as depicted, for example, in the context ofrow 75 in FIG. 4 and described in connection with FIG. 7, the pixels onthe left side may integrate more dark current than the right ones.Therefore, in order to prevent registration of any false event due todark current, the pre-determined ramp threshold (Vramp) may bedecreased/adjusted by the rate that the dark current increases along therow of pixels to compensate for the reduced level of pixel output due tothe dark current. In one embodiment, this adjusted threshold value maybe then used for a pixel to compare the level of the pixel's PIXOUTsignal. Thus, the value of the threshold voltage (Vramp) may be variableand individually-programmable for each ADC. In one embodiment, allpixels associated with a specific ADC may have the same Vramp value. Inanother embodiment, each pixel may have a pixel-specific Vramp valueprogrammable in the corresponding ADC.

It is observed here that when a row of light spots is scanned along thesurface of the object, two or more different spots from the objectscanned may be imaged on the same pixel. The spots may be in the samescanning line or may be on adjacent scanning lines. When multiple spotsare scanned across the surface of the object, such overlapping imagingmay negatively affect the correlation of the spots and the pixel ONevents and, hence, may cause ambiguity in the depth measurements. Forexample, it is seen from Eq. (1) that the depth measurement is relatedto the scan angle θ and the pixel location of the imaged light spot, asgiven by the parameter q in Eq. (1). Thus, if the scan angle is notcorrectly known for a given light spot, the depth calculation may beinaccurate. Similarly, if two or more light spots have the same value ofq, the depth calculations may become ambiguous as well. The time-stampbased approach according to particular embodiments disclosed herein maybe used to maintain the correct correlation between the pixel locationof a captured light spot and the corresponding scan angle of the lasersource. In other words, a timestamp may represent an association betweenthe values of parameters q and θ. Thus, if two spots land on the samepixel or column (from the data output point of view), thetime-to-digital conversion in the timestamping approach may allow theimaging system, here, the digital processing block 167 (FIG. 6B), toestablish a temporal correlation between these two spots to identifywhich light spot was received first in time. Such correlation may not beeasily possible in systems that do not use timestamping, such as, theearlier-described stereo-vision systems or the systems using thestructured-light approach. As a result, such systems may need to performa lot of data searching and pixel-matching to solve the correspondenceproblem.

In one embodiment, when multiple light spots are imaged by the samepixel, timestamps of the light spots may be compared to identify theearliest-received light spot and the distance may be calculated for thatlight spot only, while ignoring all subsequently-received light spots atthe same pixel. Thus, in this embodiment, the timestamp of theearliest-received light spot may be treated as the pixel-specific outputfor the corresponding pixel. Alternatively, in another embodiment, thedistance may be calculated for the list spot that is received the lastin time, while ignoring all other light spots imaged by the same pixel.In either case, any light spot received between the first or the lastlight spot may be ignored for depth calculations. Mathematically, thescan times of light spots projected by a light source may be given ast(0), t(1), . . . , t(n), in which t(i+1)−t(i)=d(t) (constant). Thepixel/column outputs may be given as a(0), a(1), . . . , a(n), which aretimestamps for the ON events and a(i) is always after t(i), but beforea(i+1). If a(i) and a(k) (i≠k) happen to be associated with the samepixel/column, only one of them may be saved as described before toremove any ambiguity in depth calculations. Based on the timerelationship between the scan time and the output time (represented bytimestamps), the processing unit, such as the digital block 167, maydetermine which output point(s) is missing. Although the processing unitmay not be able to recover the missing location, the depth calculationsfrom the available output points may suffice to provide an acceptable 3Ddepth profile of the object. It is noted here that, in one embodiment,it also may be possible for two different pixels to image a respectiveportion of the same light spot. In that embodiment, based on thecloseness of the values of the timestamp outputs from these two pixels,the processing unit may infer that a single light spot may have beenimaged by two different pixels. To resolve any ambiguity, the processingunit may use the timestamps to find an average of the respectivelocation values q, and use that average value of q in Eq. (1) tocalculate the 3D depth for such a shared light spot.

FIG. 8 depicts an example look-up table (LUT) 215 to show how an LUT maybe used in particular embodiments disclosed herein to determine 3D-depthvalues. The LUT-based approach may be used in place of theearlier-described triangulation-based depth calculations on-the-flyusing the Eq. (1). The LUT 215 lists the parameters θ, q, and Z for ascan line S_(R). The relation among these parameters is given by Eq.(1). The LUT 215 may be pre-populated with the values of theseparameters for multiple scan lines, of which only the scan line S_(R) isindicated in FIG. 8. The pre-populated LUT 215 may be stored in thesystem memory 20 (FIGS. 1 and 2), in the internal memory (not shown) ofthe processor 19, or within the digital processing block 167 (FIG. 6A).Initially, to populate the LUT 215, a light spot along scan line S_(R)may be projected at a reference distance Z_(i), for example, 1 meter,and using a specific scan angle θ_(i). These pre-determined values ofZ_(i) and θ_(i) may be used in Eq. (1) to obtain a corresponding valueof q_(i), which would indicate the column/pixel at which the imaged spotshould appear for the scan line S_(R). Different values of Z_(i) andθ_(i) may be used to obtain corresponding values of q_(i). If there is aΔZ difference between the actual and pre-determined values of Z_(i) fora light spot in scanning line S_(R), the corresponding column/pixelshould move by Δq. The values in the LUT 215 may be thus adjusted asnecessary. In this manner, for each scanning line S_(R), the LUT 215 maybe pre-populated with depth values Z_(i) as a function of θ_(i) andq_(i) using the triangulation Eq. (1). As noted before, thepre-populated LUT may be stored in the device 15. During operation, theactual values of θ_(i) and q_(i) for each light spot in a scan line oflight spots projected on a user-selected 3D object may be used as inputsto an LUT, like the LUT 215, to look-up the corresponding value Z_(i).The processor 19 or the digital block 167 may be configured to performsuch look-ups. Thus, in particular embodiments, the 3D profile of theobject may be generated by interpolating into an LUT that has beencalibrated using triangulation.

It is observed from the foregoing description that the timestamp-based3D-depth measurement using triangulation according to particularembodiments disclosed herein allows an ADC to be operated as a binarycomparator with a low resolution of just a single bit, thereby consumingsignificantly less switching power in the ADC and, hence, conserving thesystem power. A high bit resolution ADC in traditional 3D sensors, onthe other hand, may require more processing power. Furthermore,timestamp-based ambiguity resolution may also save system power incomparison with traditional imaging approaches that require significantprocessing power to search and match pixel data to resolve ambiguities.The latency may be reduced as well because all depth measurements may beperformed in one pass due to imaging/detection of all point-scannedlight spots in a single imaging step. In particular embodiments, eachpixel in the pixel array may be a single storage pixel and, hence, maybe made as small as 1 micrometer (μm) in size. In a single-storage pixeldesign, there may be only one photodiode and one junction capacitor perpixel (like the transistor 111 in FIG. 5) to integrate and storephotoelectrons. On the other hand, a pixel that has one photodiode withmultiple capacitors to store photoelectrons coming at different timesmay not be reduced to such a small size. Thus, the low-power 3D-imagingsystem with small sensors as per particular embodiments disclosed hereinmay facilitate its easy implementation in mobile applications such as,in cameras in smartphones or tablets.

FIG. 9 depicts an exemplary flowchart 220 showing how the same imagesensor, such as the image sensor unit 24 in FIGS. 1 and 2, may be usedfor both 2D imaging and 3D-depth measurements according to the subjectmatter disclosed herein. As previously mentioned, for example, the imagesensor may be part of a camera system on a mobile phone, smartphone,laptop computer, or tablet, or as part of a camera system in anindustrial robot or VR equipment. In particular embodiments, there maybe a mode switch on the device to allow a user to select between thetraditional 2D-camera mode or the 3D-imaging mode using depthmeasurements as previously described. In the traditional 2D-camera mode,in particular embodiments, the user may capture color (RGB) images orsnapshots of a scene or a particular 3D object within the scene. In the3D mode, however, the user may be able to generate a 3D image of theobject based on the camera system performing the point scan-based depthmeasurements as previously described, or by performing a sheet scan asdescribed later. In either the 2D-mode or the 3D-mode, the same imagesensor may be used in its entirety to carry out the desired imaging. Inother words, each pixel in the image sensor may be used for either a2D-imaging application or a 3D-imaging application. Such dual-modeoperation using the same image sensor may be accomplished as depicted inFIG. 9.

Various steps depicted in FIG. 9 may be performed by a single module ora combination of modules or system components in the system 15. In thedescription herein, by way of an example only, specific tasks may bedescribed as being performed by specific modules or system components.Other modules or system components may be suitably configured to performsuch tasks as well.

As depicted in FIG. 9, at block 222, the image sensor may be provided tocapture a 2D image of a 3D object that is illuminated by ambient light.An example 3D object 26 is depicted in FIGS. 2-4. The image sensor mayhave a plurality of pixels arranged in a 2D array such as, the 2D pixelarray 42 shown in FIGS. 2 and 5. At block 224, a laser light source,such as the laser light source 33 or the light source module 22 in FIG.2, may be provided to illuminate the 3D object using a point scan or asheet scan with light from the laser source. The light from the lasersource may be in addition to the ambient light. The point scan approachhas been described earlier with reference to FIGS. 3 and 4. A sheet scanwill be described in connection with FIGS. 12-15. Thereafter, at block226, the depth of the 3D object, which is now illuminated by the ambientlight as well as the laser light, may be determined using at least onerow of pixels in the image sensor. The triangulation-based depthmeasurements may be made by using timestamping according to particularembodiments disclosed herein. Thus, the general approach outlined inFIG. 9, may allow a device, such as the device 15 in FIGS. 1 and 2, tobe configured and operated for 2D as well as 3D imaging without the needfor separate image sensors.

FIG. 10 depicts a timing diagram 230 that shows example timing ofdifferent signals in the system 15 of FIGS. 1 and 2 to generate a 2Dimage using a 2D-linear mode of operation according to the subjectmatter disclosed herein. It is noted here that the 2D image may be anRGB image of a scene or a 3D object within the scene under ambient lightillumination, which may include occasional use of a camera flash orother similar component (not shown). In contrast to the3D-imaging-related embodiments in FIGS. 7 and 11, however, there may notbe any illumination by the laser light source 33 (FIG. 2) in case of the2D imaging in the embodiment of FIG. 10. Many signals shown in FIG. 10are also indicated in FIG. 7. In view of the earlier detaileddescription of FIG. 7, only the salient aspects of FIG. 10 are describedherein. It is noted here that the control signals RSEL, RST, TX, RAMP,and ADC_RST indicated in FIG. 10 are for the row of pixels containingpixels 106-108 in FIG. 5 and, hence, for ease of description, thesesignals are identified using the same reference numerals as those usedin FIG. 7, despite the difference in waveforms and timing of the signalsin FIGS. 7 and 10. Furthermore, the depiction in FIG. 10 is for a singlepixel, here the pixel 108 in FIG. 5. Therefore, the PIXOUT signal 119,the Counter Clock signal 204, and the ADCOUT signal 197 in FIG. 10 areshown using the same reference numerals as those for correspondingsignals PIXOUT1, Counter Clock-1 and ADCOUT1 in FIG. 7. The pixel output119 is generated by linearly integrating the photoelectrons collected bythe pixel 108 over a pre-determined time period. As before, thedescription of FIG. 10 in the context of pixel 108 remains applicable tocorresponding signals associated with other pixels in the pixel array42.

As noted before, in particular embodiments, each column-specific ADC,such as the ADC unit 175 in FIG. 6B, may be a single-slope ADC. As incase of FIG. 7, pixels in the same row may be selected and resettogether, as shown by the RSEL signal 122 and the RST signal 128 in FIG.10. The column ADCs may be reset also using the common ADC_RST signal192. In FIG. 10, the reset state of the ADC associated with pixel 108 isindicated by the field 234 in the ADCOUT signal 197. After the pixel 108and its ADC are reset, a threshold or reference voltage level may beenabled as shown by the voltage level 236 for the Vramp signal 143. Theramp then ramps down from this voltage level 236 to digitize thecomparator offset of the ADC unit, as given by the field 238 in theADCOUT signal 197. In one embodiment, the clock pulses in the counterclock 204 may be used to generate a count value as the offset 238. Theclock pulses may be counted from the time the Vramp signal 143 reachesthe threshold level 236 until it drops to the reset level of the pixeloutput, here, the PIXOUT signal 119. Thereafter, the respective transfer(TX) line 132 may be pulsed to trigger the transfer of chargeaccumulated on the photodiode 110 to the floating diffusion 111 forreadout. While the TX pulse is asserted, the Vramp signal 143 may riseto the threshold level 236 and a counter in the pixel-specific ADC, suchas the counter 181 in FIG. 6B, may be initialized with an invertedoffset value as indicated by the field 240. The inverted offset value240 may represent the negative of the offset value 238. After the TXpulse 132 is de-asserted, the ADC unit for the pixel 108 may startdigitizing the received pixel signal (PIXOUT) until the Vramp threshold143 drops to the level of the PIXOUT signal 119. This operation isillustrated by the up count field 242 in the ADCOUT signal 197. Thecount value 242 may be based on the clock pulses of the counter clock204 and may represent a combined value including the offset count (atfield 238) and the pixel-specific portion of the image signal for pixel108, as depicted using the reference numeral 243. A comparator (notshown) in the ADC unit may compare the comparator offset value digitizedat field 238 against the up count value 242. Thus, in one embodiment,the RGB image signal 244 may be obtained by adding the ADC values in thefields 240 and 242, thereby, effectively removing the offset value 238from the combined value (offset+signal) in the up count field 242.

The operation depicted in FIG. 10 may be performed for each pixel in thepixel array 42. Each column ADC may generate a corresponding RGB imagesignal in the form of a multi-bit output from the ADC-based counter,such as the counter 181 in FIG. 6B. The multi-bit output, like theoutput at reference numeral 244 in FIG. 10, may be needed to effectivelyrepresent the color content of the image signal. The RGB image signaloutputs from the ADCs in the column ADC unit 153 may be collectivelyrepresented by the Dout signal 140 (FIGS. 6A and 6B), which may beprocessed by the digital block 167 to present the 2D color image of thescene via the MIPI interface 168.

Additional details of the 2D imaging and related waveforms depicted inFIG. 10 may be obtained from the U.S. Pat. No. 7,990,304 issued on Aug.2, 2011 to Lim et al. The 2D-imaging related disclosure in the Lim etal. patent is related to the present disclosure and is incorporatedherein by reference in its entirety.

FIG. 11 depicts a timing diagram 250 that shows example timing ofdifferent signals in the system 15 of FIGS. 1 and 2 to generatetimestamp-based pixel-specific outputs in a 3D-logarithmic (log) mode ofoperation according to the subject matter disclosed herein. Aspreviously mentioned, the 3D-depth measurements may be performed using a3D-linear mode or a 3D-logarithmic mode depending on the level ofambient light. Furthermore, during the 3D-depth measurements, a 3Dobject, such as the 3D object 26 in FIG. 2, may be illuminated by theambient light as well as by the visible light (or other light, such as,NIR light) from the laser scan. Therefore, the 3D-logarithmic mode maybe used for depth measurements if ambient light is too strong to berejected by the 3D-linear mode. In view of the CDS-based imaging toremove the offset or other noise from the final image signal, alogarithmic mode may not be needed for the 2D-imaging-related waveformsdepicted in FIG. 10. In the case of the 3D-depth measurements, however,a strong ambient light may interfere with the light from the laser lightsource during point scans. In the 3D-linear mode of operation, suchinterference may overwhelm or suppress the visible/NIR light reflectedfrom a point-scanned light spot and, hence, may result in an inaccuratedetection of the light received from the light spot. Thus, in particularembodiments, it may be desirable to reject the pixel charge attributableto the ambient light if the intensity of the ambient light is sensed tobe above a pre-determined illuminance level (or intensity threshold),such as, for example, 10000 (10K) lux. Such ambient light rejection maybe accomplished using the 3D-log mode of operation depicted in FIG. 11.

As before, the same reference numerals are used in FIGS. 7, 10, and 11to refer to the similarly-named signals (or signals having similarfunctionality) and also for ease of description. It is understood,however, that the signals shown in FIGS. 7, 10 and 11 relate to specificmodes of imaging. Thus, for example, the timing diagram 230 depicted inFIG. 10 depicts a specific relationship among the signals shown thereinwhen a user selects a 2D color imaging mode of operation. Thesimilarly-named signals in FIGS. 7 and 11, however, relate to a3D-imaging mode of operation and, hence, may have different timingrelationships. Furthermore, even between FIGS. 7 and 11, some signalsmay differ in waveforms because FIG. 7 relates to a 3D-linear mode ofoperation, whereas FIG. 11 relates to a 3D logarithmic mode ofoperation. In view of the earlier detailed description of FIG. 7, onlythe salient aspects of FIG. 11 are described herein. Like FIG. 7, thetiming diagram 250 in FIG. 11 is also with reference to pixels 107 and108 in FIG. 5. The description of FIG. 11 remains applicable to allother pixels in the pixel array 42.

In the 3D-linear mode, the pixel-specific output may be generated bylinearly integrating the photoelectrons collected by the pixel over apre-determined time period. Thus, in the linear mode, an output voltageof a pixel may be proportional to the total photons collected/integratedover a given time period. In the 3D-log mode, however, thepixel-specific output may be proportional to the natural logarithm of aninstantaneous photo-current produced by the pixel during thepre-determined time period upon detecting the laser light reflected fromthe 3D object. Mathematically, the photo current generated by aphotodiode, such as the PPD 110 in FIG. 5, may be represented by thefollowing relationship:

$\begin{matrix}{I_{ph} \propto e^{\frac{V_{ph}}{V_{T}},}} & (2)\end{matrix}$

in which I_(ph) is the photocurrent of the diode, V_(ph) is the voltageacross the diode, and V_(T) is the thermal voltage. Thus, V_(ph) and,hence, the respective pixel output (PIXOUT) may be made proportional tothe natural logarithm of the instantaneous diode current I_(ph), suchas, if ambient light rejection is desired. As noted before, heavyambient light may restrict photon collection if linear integration isdone. Thus, in such circumstances, the sensing of instantaneousphotocurrent using the 3D-log mode may be more desirable.

In particular embodiments, the device 15 may include an ambient lightsensor (not shown). The processor 19 or the digital block 167 may beconfigured to sense the ambient light intensity as soon as the3D-imaging mode is selected by the user to determine whether to use the3D-linear mode or the 3D-log mode. In one embodiment, the ambient lightlevel may be sensed substantially simultaneously with the assertion ofan RSEL signal, which may indicate the initiation of the imaging of thelight reflected from the point-scanned light spots. In anotherembodiment, the ambient light level may be sensed substantiallysimultaneously with the initiation of the visible light point scan bythe laser source. Based on the level of the ambient light, the processor19 or the digital block 167 may choose either the 3D-linear mode or the3D-log mode of depth measurements. In a still further embodiment, theambient light level may be sensed periodically and continuously during a3D-depth measurement. In that case, the 3D-mode of operation may beswitched from linear to logarithmic, and vice versa, at any time priorto or during an ongoing imaging operation. Other approaches for sensingthe ambient light level may be suitably devised.

Referring now to the embodiment of FIG. 11, it is observed that, in the3D-logarithmic mode, the row-specific RST signal 128 may be asserted (orturned on “high”) and may remain high/asserted for the selected rowduring the entire period of generation of the pixel output. In contrast,in the 3D linear mode of FIG. 7, the RST signal 128 may be initiallyasserted (or turned on “high”) to reset the pixels in the row to apre-determined voltage level, but later turned off (or de-asserted)during linear integration of the photoelectrons. The TX signal 132,however, may remain high, like in case of the 3D linear mode of FIG. 7.Thus, in particular embodiments, the appropriate level of the RST signalmay be used to select the linear mode versus the logarithmic mode. Inthe logarithmic mode, in one embodiment, after the ADCs associated withpixels 107-108 are reset using the ADC_RST signal 192, these ADCs mayinitially sample the ambient level to enable the ADCs to appropriatelyaccount for the signal levels of the pixel output (PIXOUT) signals whenthey are received. After ADCs are reset, the RAMP threshold 143 may beenabled, and the ADC counters may enter a wait state to wait for an “ONevent” to occur at the respective pixel. When a pixel receives incidentlight (reflected from a projected light spot), its PIXOUT signal maystart drooping. In contrast to the linear drop in FIG. 7, the PIXOUTsignals 118-119 in FIG. 11 may exhibit short, instantaneous drops252-253, respectively, which reflect the instantaneous photo-currentproduced by the respective detection by the pixels of the reflectedvisible light. When the PIXOUT signals 118-119 reach the pre-determinedVramp threshold 143, the ADC counters may start counting. All countersmay stop simultaneously, after a pre-determined time for chargeintegration is over, as given by the transition of the RAMP signal 143to its “high” state and as indicated by the dotted line 255. The countedvalues are indicated by the data field 257 of ADCOUT1 and the data field259 of the ADCOUT2 signals for pixels 108 and 107, respectively. Thecount values in the logarithmic mode may be different from those in thelinear mode and, hence, different reference numerals are used for the“up count” fields in the ADCOUT signals in FIGS. 7 and 11. As in case ofFIG. 7, pixel scanned later may have a smaller count value for its ADCoutput than the one that is scanned earlier.

As previously mentioned in connection with FIG. 7, instead of per columnup-counters, down counters may be used in the ADC units in theembodiments of FIGS. 10 and 11. Similarly, a global counter basedapproach may be implemented instead of individual ADC-specific counters.

Thus, as previously described, the same image sensor (and all of thepixels in its pixel array) may be used as per teachings of the presentdisclosure for routine 2D imaging as well as for 3D-depth measurements.In the 2D mode, the sensor may work in the linear mode as a regular 2Dsensor. During the 3D-depth measurements, however, the sensor mayoperate in a linear mode under moderate ambient light, but may switch toa logarithmic mode of signal detection under strong ambient light to beable to use the visible (or NIR) light source. Thus, the imagingapproaches described herein may be compatible with existing 2D-sensordesigns because the same 4T PPD pixel may be used for both 2D and 3Dimaging. This allows for the sensor design to be small in size (withsmaller pixels), more versatile, and operable at low power. Theseattributes, in turn, save space and cost for mobile devices containingsuch an image sensor. Furthermore, in consumer mobile devices andcertain other applications, the usage of visible light laser (inaddition to the ambient light) for 3D-depth measurements may be betterfor eye safety than conventional NIR sensors. At visible spectrum, thesensor may have higher quantum efficiency than at the NIR spectrum,leading to lower power consumption of the light source, which, in turn,conserves power in the mobile devices.

FIG. 12 depicts another example embodiment of an image sensor 1200 thatmay make 3D-depth measurements using a sheet scan according to thesubject matter disclosed herein. To relax the requirements related toscanner-sensor alignment that may be associated with epipolar scanning,the image sensor 1200 provides a sheet scan as opposed to a point scanto provide 3D-depth measurements, Additionally, scanning only one row ata time to make an epipolar point-scan measurement may introduceinterference in the 2D image measurement in adjacent rows. All rows ofthe pixel array of the image sensor 1200 are simultaneously operative torecord timestamps of laser events. Disparity of each laser dot in, forexample, a horizontal direction may be determined by using itscorresponding timestamp in the same row. An entire depth map may begenerated in one pass of a laser sheet scan.

The image sensor 1200 may include many of the same components andmodules as the system 15 depicted in FIGS. 1 and 2, the pixel array 42depicted in FIG. 5, the image sensor unit 24 depicted in FIG. 6A, andthe CDS+ADC unit 175 depicted in FIG. 6B, although not depicted in FIG.12. Additionally, the various components and modules of the image sensor1200 may operate in a manner that is the same or is similar to theoperation of the components and modules of the system 15 depicted inFIGS. 1 and 2, the pixel array 42 depicted in FIG. 5, the image sensorunit 24 depicted in FIG. 6A, and the CDS+ADC unit 175 depicted in FIG.6B. Further, the parameters d, h, q, θ, and Z for the triangulation Eq.(1) are indicated in FIG. 12.

The image sensor 1200 may include a light source 1201 and an imagesensor unit 1202. The light source 1201 may project a line of light 1203across a view of view 1204 of the light source. The field of view 1204may additionally or alternatively be considered to be a field of view1204 for the image sensor unit 1202 (and/or for a pixel array 1205 ofthe image sensor unit 1202). The line of light 1203 is scanned 1206across the field of view 1204 in a direction that is substantiallyperpendicular to the direction of the line of light 1203. The scanningdirection may be either towards to the right, as depicted in FIG. 12, ortowards the left. The scanning of the line of light 1203 may beconsidered to be a sheet scan of the field of view 1204.

A portion of the line of light 1203 will be reflected of an object (notshown) in the field of view 1204. The reflected portion of the line oflight will be received by the image sensor unit 1202. The image sensorunit may include a lens 1207 and the pixel array 1205. The receivedportion of the line of light will be incident upon the pixel array 1205,as indicated at 1208. As the light of light 1203 is scanned across thefield of view 1204, reflected light will be incident on the respectivecolumns of pixels of the pixel array 1205 corresponding to the scanningmotion. A controller (not shown in FIG. 12, but corresponding to pixelarray control and processing circuits 46 in FIG. 2) selectively enablesthe pixels in the respective columns of the pixel array 1205 insynchronism with the scanning motion 1206.

It will be understood that a complete received line of light is depictedin FIG. 12 for convenience even though only a portion of the line ofline 1203 may be reflected and may be incident upon the pixel array1205. It will also be understood that although the line of light 1203 isdepicted as being a vertically oriented line of light, the line of lightcould alternatively be a horizontally oriented line of light, in whichcase the scanning direction would either be in an upward direction or adownward direction with respect to FIG. 12, and rows of the pixel array1205 would be synchronized with the scanning motion. In the case inwhich the light of light is vertically oriented, the baseline ishorizontal. In the case in which the light of light is horizontallyoriented, the base line is vertical. The scanning direction is parallelto the baseline.

FIG. 13 depicts an example LUT 1300 that may be used to determine3D-depth values for a sheet scan. The LUT-based approach may be used inplace of the earlier-described triangulation-based depth calculationson-the-fly using the Eq. (1). The LUT 1300 lists the parameterstimestamp t, θ, q, and Z for a sheet scan. The relation among theseparameters is given by Eq. (1). In a 3D mode, the timestamp for eachpixel is obtained and using the timestamp, the scan angle θ may bedetermined. From the scan angle θ, q and Z may be determined. Morespecifically, by comparing q and the column number corresponding to thecurrent timestamp, the real Z may be determined.

FIG. 14 depicts a block diagram of an example embodiment of a pixelarray 1205 and of associated processing circuits according to thesubject matter disclosed herein. The pixel array 1205 may include manyof the same components as the 2D pixel array 42 depicted in FIG. 5.Additionally, the components of the pixel array 1205 may operate in asame or similar manner as the components of the pixel array 42. Similarto the configuration of the 2D pixel array 42, the pixel array 1205includes nine pixels 100′-108′ arranged as a 3×3 array for ease ofdescription only. In practice, the pixel array 1205 may contain hundredsof thousands or millions of pixels in multiple rows and columns. In oneembodiment, each pixel 100′-108′ may have an identical or nearly anidentical configuration as depicted in FIG. 12. The constituent circuitelements of only pixel 108′ in FIG. 12 are labeled with the samereference numerals as pixel 108 in FIG. 5. The operation of pixels100′-108′ in FIG. 12 may be the same as the operation of the pixels100-108 in FIG. 5.

The pixel array 1205 may differ from the pixel array 42 in that theoutput of each pixel 100′-108′ is input to a separate ADC and CDScircuit in an array of ADCs and CDS circuits 1503. Time multiplexing isused to obtain 2D-imaging data and 3D-depth data. In both the 2D-imagemode and the 3D-depth mode, each pixel output is separate from the otherpixel outputs in the same column. For example, the pixel outputs in theleft-most column of the pixel array 1205 are indicated as pixel outputs1406 a-1406 c. The pixel outputs in the center column of the pixel array1205 are indicated as pixel outputs 1406 d-1406 f, and the pixel outputsin the right-most column of the pixel array 1205 are indicated as pixeloutputs 1406 h-1406 i. The column decoder 145 may be used to synchronizethe respective columns when the sheet scan involves a verticallyoriented line of line 1203 and a horizontal sheet scan. Anotherdifference between the embodiment of the pixel array 1205 and theassociated processing circuits of FIG. 12 and the embodiment of thepixel array 42 and the associated processing circuits of FIG. 5 is thatin one embodiment there is a separate ADC and CDS for each pixel outputin FIG. 12 as opposed for each column output in FIG. 5.

In an alternative embodiment, the output of a group of pixels may becoupled together and the grouped output may be input to a separate ADC(and CDS circuit). For example, the nine pixel outputs of the 3×3 pixelarray 1205 may be coupled together and input to an ADC. Other groupingsof pixels are possible, such as a 2×2 pixel grouping. Although the3D-depth resolution may be reduced by grouping outputs together, theresolution of an image in the 2D-imaging mode will remain the same.

FIG. 15 depicts an example embodiment 1500 of the pixel array 1205 andthe associated processing circuits of FIG. 12 according to the subjectmatter disclosed herein. The example embodiment 1500 includes the pixelarray 1205 on a first die 1501 that is positioned above a second die1502. The second die 1502 may include the ADCs and some of theassociated processing circuits of FIG. 12. In one embodiment, the seconddie 1502 includes an ADC array 1503, a row driver array 1504, and a biasand other circuitry region 1505.

The pixel array 1205 includes a plurality of pixels arranged in rows andcolumns. The output 1506 of a pixel is coupled to a corresponding ADC1507. It will be understood that the output 1506 of only one pixel isindicated in FIG. 15 as being coupled to a corresponding ADC forconvenience. In one embodiment, the ADC array 1504 may include CDScircuitry (not indicated in FIG. 15) for each ADC 1507. In anotherembodiment, the pixel array 1205 and all or some of the associatedprocessing circuits for the 3D-mode may be located all on the same die1501.

FIG. 16 depicts an example overall layout of the system 15 in FIGS. 1and 2 according to the subject matte disclosed herein. Hence, for easeof reference and description, the same reference numerals are used inFIGS. 1, 2 and 12 for the common system components/units.

As previously described, the imaging module 17 may include the hardwaredepicted in the example embodiments of FIGS. 2, 5, 6A and 6B toaccomplish 2D imaging and 3D-depth measurements according to the subjectmatter disclosed herein. The processor 19 may be configured to interfacewith a number of external devices. In one embodiment, the imaging module17 may function as an input device that provides data inputs in the formof pixel event data such as, the processed data output 170 in FIG. 6A,to the processor 19 for further processing. The processor 19 may alsoreceive inputs from other input devices (not shown) that may be part ofthe system 15. Some examples of such input devices include a computerkeyboard, a touchpad, a touch-screen, a joystick, a physical or virtualclickable button, and/or a computer mouse/pointing device. In FIG. 16,the processor 19 is depicted coupled to the system memory 20, aperipheral storage unit 265, one or more output devices 267, and anetwork interface unit 268. In FIG. 16, a display unit is depicted as anoutput device 267. In some embodiments, the system 15 may include morethan one instance of the devices depicted. Some examples of the system15 include a computer system (desktop or laptop), a tablet computer, amobile device, a cellular phone, a video gaming unit or console, a M2Mcommunication unit, a robot, an automobile, a virtual reality equipment,a stateless “thin” client system, a dash-cam or rearview camera systemof a vehicle, or any other type of computing or data processing device.In various embodiments, all of the components depicted in FIG. 16 may behoused within a single housing. Thus, the system 15 may be configured asa standalone system or in any other suitable form factor. In someembodiments, the system 15 may be configured as a client system ratherthan a server system.

In particular embodiments, the system 15 may include more than oneprocessor (e.g., in a distributed processing configuration). If thesystem 15 is a multiprocessor system, there may be more than oneinstance of the processor 19 or there may be multiple processors coupledto the processor 19 via their respective interfaces (not shown). Theprocessor 19 may be a system on chip (SoC) and/or may include more thanone CPU.

As previously mentioned, the system memory 20 may be anysemiconductor-based storage system such as, for example, DRAM, SRAM,PRAM, RRAM, CBRAM, MRAM, STT-MRAM, and the like. In some embodiments,the memory unit 20 may include at least one 3DS memory module inconjunction with one or more non-3DS memory modules. The non-3DS memorymay include Double Data Rate or Double Data Rate 2, 3, or 4 SynchronousDynamic Random Access Memory (DDR/DDR2/DDR3/DDR4 SDRAM), or Rambus®DRAM, flash memory, various types of Read Only Memory (ROM), etc. Also,in some embodiments, the system memory 20 may include multiple differenttypes of semiconductor memories, as opposed to a single type of memory.In other embodiments, the system memory 20 may be a non-transitorydata-storage medium.

The peripheral storage unit 265, in various embodiments, may includesupport for magnetic, optical, magneto-optical, or solid-state storagemedia such as hard drives, optical disks (such as compact disks (CDs) ordigital versatile disks (DVDs)), non-volatile random access memory (RAM)devices, and the like. In some embodiments, the peripheral storage unit265 may include more complex storage devices/systems such as disk arrays(which may be in a suitable RAID (redundant array of independent disks)configuration) or storage area networks (SANs), and the peripheralstorage unit 265 may be coupled to the processor 19 via a standardperipheral interface such as a small computer system interface (SCSI)interface, a Fibre Channel interface, a Firewire® (IEEE 1394) interface,a peripheral component interface express (PCI Express™) standard basedinterface, a universal serial bus (USB) protocol based interface, oranother suitable interface. Various such storage devices may benon-transitory data-storage media.

The display unit 267 may be an example of an output device. Otherexamples of an output device include a graphics/display device, acomputer screen, an alarm system, a CAD/CAM (computer aideddesign/computer aided machining) system, a video game station, asmartphone display screen, or any other type of data output device. Insome embodiments, the input device(s), such as the imaging module 17,and the output device(s), such as the display unit 267, may be coupledto the processor 19 via an I/O or peripheral interface(s).

In one embodiment, the network interface 268 may communicate with theprocessor 19 to enable the system 15 to couple to a network (not shown).In another embodiment, the network interface 268 may be absentaltogether. The network interface 268 may include any suitable devices,media and/or protocol content for connecting the system 15 to a network,whether wired or wireless. In various embodiments, the network mayinclude local area networks (LANs), wide area networks (WANs), wired orwireless Ethernet, telecommunication networks, or other suitable typesof networks.

The system 15 may include an on-board power supply unit 270 to provideelectrical power to various system components depicted in FIG. 16. Thepower supply unit 270 may receive batteries or may be connectable to anAC electrical power outlet. In one embodiment, the power supply unit 270may convert solar energy into electrical power.

In one embodiment, the imaging module 17 may be integrated with ahigh-speed interface such as, for example, a universal serial bus 2.0 or3.0 (USB 2.0 or 3.0) interface or above, that plugs into any personalcomputer (PC) or laptop. A non-transitory, computer-readabledata-storage medium, such as, the system memory 20 or a peripheral datastorage unit such as a CD/DVD may store program code or software. Theprocessor 19 and/or the digital processing block 167 (FIG. 6A) in theimaging module 17 may be configured to execute the program code, wherebythe device 15 may be operative to perform the 2D imaging and 3D-depthmeasurements as previously described such as, the operations describedearlier with reference to FIGS. 1-15. The program code or software maybe proprietary software or open source software which, upon execution bythe appropriate processing entity, such as the processor 19 and/or thedigital block 167, may enable the processing entity to capture pixelevents using their precise timing, process them, render them in avariety of formats, and display them in the 2D and/or 3D formats. Asnoted earlier, in certain embodiments, the digital processing block 167in the imaging module 17 may perform some of the processing of pixelevent signals before the pixel output data are sent to the processor 19for further processing and display. In other embodiments, the processor19 may also perform the functionality of the digital block 167, in whichcase, the digital block 167 may not be a part of the imaging module 17.

In the preceding description, for purposes of explanation and notlimitation, specific details are set forth (such as particulararchitectures, waveforms, interfaces, techniques, etc.) in order toprovide a thorough understanding of the disclosed technology. It will,however, be apparent to those skilled in the art that the disclosedtechnology may be practiced in other embodiments that depart from thesespecific details. That is, those skilled in the art will be able todevise various arrangements which, although not explicitly described orshown herein, embody the principles of the disclosed technology. In someinstances, detailed descriptions of well-known devices, circuits, andmethods are omitted so as not to obscure the description of thedisclosed technology with unnecessary detail. All statements hereinreciting principles, aspects, and embodiments of the disclosedtechnology, as well as specific examples thereof, are intended toencompass both structural and functional equivalents thereof.Additionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents developed in thefuture, such as, for example, any elements developed that perform thesame function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the artthat block diagrams herein (e.g., in FIGS. 1 and 2) can representconceptual views of illustrative circuitry or other functional unitsembodying the principles of the technology. Similarly, it will beappreciated that the flow chart in FIG. 9 represents various processeswhich may be substantially performed by a processor (e.g., the processor19 in FIG. 12 and/or the digital block 167 in FIG. 6A). Such processormay include, by way of example, a general-purpose processor, aspecial-purpose processor, a conventional processor, a digital signalprocessor (DSP), a plurality of microprocessors, one or moremicroprocessors in association with a DSP core, a controller, amicrocontroller, application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs) circuits, any other type of integratedcircuit (IC), and/or a state machine. Some or all of the functionalitiesdescribed herein in the context of FIGS. 1-15 also may be provided bysuch processor, in the hardware and/or software.

When certain inventive aspects require software-based processing, suchsoftware or program code may reside in a computer-readable data-storagemedium. As noted earlier, such data storage medium may be part of theperipheral storage 265 or may be part of the system memory 20 or theinternal memory (not shown) of processor 19. In one embodiment, theprocessor 19 or the digital block 167 may execute instructions stored onsuch a medium to carry out the software-based processing. Thecomputer-readable data-storage medium may be a non-transitorydata-storage medium containing a computer program, software, firmware,or microcode for execution by a general-purpose computer or a processormentioned above. Examples of computer-readable storage media include,but are not limited to, a ROM, a RAM, a digital register, a cachememory, semiconductor memory devices, magnetic media such as internalhard disks, magnetic tapes and removable disks, magneto-optical media,and optical media such as CD-ROM disks and DVDs.

Alternative embodiments of the imaging module 17 or the system 15including such an imaging module according to the subject matterdisclosed herein may include additional components responsible forproviding additional functionality, including any of the functionalityidentified above and/or any functionality necessary to support thesolution as per the subject matter disclosed herein. Although featuresand elements are described above in particular combinations, eachfeature or element can be used alone without the other features andelements or in various combinations with or without other features. Asmentioned before, various 2D and 3D-imaging functions described hereinmay be provided through the use of hardware (such as circuit hardware)and/or hardware capable of executing software/firmware in the form ofcoded instructions or microcode stored on a computer-readabledata-storage medium (mentioned above). Thus, such functions andillustrated functional blocks are to be understood as being eitherhardware-implemented and/or computer-implemented, and thusmachine-implemented.

The foregoing describes a system and method in which the same imagesensor, that is, all of the pixels in the image sensor, may be used tocapture both a 2D image of a 3D object and 3D depth measurements for theobject. The image sensor may be part of a camera in a mobile device suchas, a smartphone. A laser light laser source may be used to point scanthe surface of the object with light spots, which may be then detectedby a pixel array in the image sensor to generate the 3D depth profile ofthe object using triangulation. In the 3D mode, the laser may project asequence of light spots on the surface of the object along a scan line.The illuminated light spots may be detected using a row of pixels in thepixel array such that the row forms an epipolar line of the scan line.The detected light spots may be timestamped to remove any ambiguity intriangulation and, hence, to reduce the amount of depth computation andsystem power. A timestamp may also provide a correspondence between thepixel location of a captured laser spot and the respective scan angle ofthe laser light source to determine depth using triangulation. The imagesignals in the 2D mode may be represented by a multi-bit output from anADC unit in the image sensor, but the ADC unit may produce just a binaryoutput to generate timestamp values for 3D-depth measurements. To rejectstrong ambient light, the image sensor may be operated in a3D-logarithmic mode as opposed to a 3D-linear mode.

As will be recognized by those skilled in the art, the innovativeconcepts described in the present application can be modified and variedover a wide range of applications. Accordingly, the scope of patentedsubject matter should not be limited to any of the specific exemplaryteachings and disclosure described above, but is instead defined by thefollowing claims.

What is claimed is:
 1. An imaging unit, comprising: a light source thatprojects a line of light that is scanned in a first direction across afield of view of the light source, the line of light oriented in asecond direction that is substantially perpendicular to the firstdirection; and a pixel array arranged in at least one row of pixels thatextends in a direction that is substantially parallel to the seconddirection, at least one pixel in a row capable of generatingtwo-dimensional (2D) color information of an object in the field of viewof the light source based on a first light reflected from the object andcapable of generating three-dimensional (3D) depth information of theobject based on the line of light reflecting from the object, the3D-depth information comprising time-of-flight information.
 2. Theimaging unit of claim 1, further comprising a time-to-digital convertercoupled to the pixel, the time-to-digital converter generating the3D-depth information based on the pixel detecting the line of lightbeing reflected from the object.
 3. The imaging unit of claim 1, whereinthe 3D-depth information comprises timestamp information.
 4. The imagingunit of claim 1, further comprising a plurality of time-to-digitalconverters, each pixel in a row of the pixel array being coupled to acorresponding time-to-digital converter that generates the 3D-depthinformation for the pixel based on the pixel detecting the line of lightbeing reflected from the object.
 5. The imaging unit of claim 4, furthercomprising a controller coupled to the light source and the plurality oftime-to-digital converters, the controller controlling thetime-to-digital converters to be synchronized with the line of lightthat is scanned across the field of view of the light source.
 6. Theimaging unit of claim 1, further comprising a time-to-digital convertercoupled to a group of pixels capable of generating 2D-color informationand 3D-depth information, the time-to-digital converter generating the3D-depth information based on at least one pixel of the group of pixelsdetecting the line of light being reflected from the object.
 7. Theimaging unit of claim 1, wherein the first direction comprisessubstantially a horizontal direction with respect to the field of viewof the light source, and the second direction comprises substantially avertical direction with respect to the field of view of the lightsource.
 8. The imaging unit of claim 1, wherein the first directioncomprises substantially a vertical direction with respect to the fieldof view of the light source, and the second direction comprisessubstantially a horizontal direction with respect to the field of viewof the light source.
 9. An image sensor unit, comprising: a pixel arrayarranged in at least one row of pixels that extends in a firstdirection, at least one pixel in a row capable of generatingtwo-dimensional (2D) color information of an object based on a firstlight reflected from the object in a field of view of the pixel arrayand capable of generating 3D-depth information of the object based on aline of light reflecting from the object, the 3D-depth informationcomprising time-of-flight information, the light of light being orientedin a second direction that is substantially perpendicular to the firstdirection, and the line of light being scanned across the field of viewof the pixel array in substantially the first direction; and atime-to-digital converter coupled to the pixel, the time-to-digitalconverter generating the 3D-depth information based on the pixeldetecting the line of light being reflected from the object.
 10. Theimage sensor unit of claim 9, further comprising a plurality oftime-to-digital converters, each pixel in a row of the pixel array beingcoupled to a corresponding time-to-digital converter that generates the3D-depth information for the pixel based on the pixel detecting the lineof light being reflected from the object.
 11. The image sensor unit ofclaim 10, further comprising a controller coupled to the plurality oftime-to-digital converters, the controller controlling thetime-to-digital converters to be synchronized with the line of lightthat is scanned across the field of view of the pixel array.
 12. Theimage sensor unit of claim 9, wherein the 3D-depth information comprisestimestamp information.
 13. The image sensor unit of claim 9, wherein thetime-to-digital converter is coupled to a group of pixels capable ofgenerating 2D-color information and 3D-depth information, thetime-to-digital converter generating the 3D-depth information based onat least one pixel of the group of pixels detecting the line of lightbeing reflected from the object.
 14. The image sensor unit of claim 9,wherein the first direction comprises substantially a horizontaldirection with respect to the field of view of the pixel array, and thesecond direction comprises substantially a vertical direction withrespect to the field of view of the pixel array.
 15. The image sensorunit of claim 9, wherein the first direction comprises substantially avertical direction with respect to the field of view of the pixel array,and the second direction comprises substantially a horizontal directionwith respect to the field of view of the pixel array.
 16. The imagesensor unit of claim 9, further comprising a light source that projectsthe line of light that is scanned in the first direction across a fieldof view of the pixel array.
 17. A method, comprising: projecting from alight source a line of light oriented in a first direction across afield of view of a light source in a second direction that issubstantially perpendicular to the first direction; and generating at apixel two-dimensional (2D) color information of an object in the fieldof view of the light source based on a first light reflected from theobject and three-dimensional (3D) depth information of the object basedon the line of light reflecting from the object, the pixel being capableof generating 2D color information of the object and 3D-depthinformation of the object, the pixel further being part of a pixel arraythat is arranged in at least one row of pixels that extends in adirection that is substantially parallel to the second direction, thepixel being in a row of the pixel array, and the 3D-depth informationcomprising time-of-flight information.
 18. The method of claim 17,wherein a time-to-digital converter is coupled to the pixel, the methodfurther comprising: generating at the time-to-digital converter the3D-depth information based on the pixel detecting the line of lightbeing reflected from the object.
 19. The method of claim 17, wherein thefirst direction comprises substantially a horizontal direction withrespect to the field of view of the light source, and the seconddirection comprises substantially a vertical direction with respect tothe field of view of the light source.
 20. The method of claim 17,wherein the first direction comprises substantially a vertical directionwith respect to the field of view of the light source, and the seconddirection comprises substantially a horizontal direction with respect tothe field of view of the light source.